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Minnesota  Center  for  Industrial  Mathematics 
Final  Report  for  AFOSR 


OBJECTIVES 

The  objective  of  the  Minnesota  Center  for  Industrial  Mathematics  (MCIM)  was  to 
demonstrate  to  industry  the  value  of  mathematics  in  research  while  at  the  same  time 
preparing  students  for  job  opportunities  in  industry  and  government.  This  has  been 
accomplished  through  student  internships  at  industrial  sites.  The  internship,  which  lasts  3 
months,  is  co-mentored  by  an  industry  researcher  and  a  faculty  member  at  the  MCIM. 

As  an  indication  of  the  success  of  the  program,  the  majority  of  funding  for  the  interns 
was  provided  by  industry;  industry  also  provided  free  time  for  its  researchers  to  mentor 
the  students. 


STATUS  OF  EFFORT 

Below  is  a  list  of  students’  who  successfully  completed  internships  in  industry.  Financial 
support  was  provided  by  the  grant  as  well  as  from  financial  contributions  from  companies 
participating  in  the  MCIM’s  industrial  mathematics  internship  program.  These  students 
are  pursuing  Master’s  or  Ph.D.  degrees  in  Industrial  and  Applied  Mathematics.  The 
students’  theses  were  based  on  research  conducted  while  completing  the  internship. 

1.  Yi-Ju  Chao,  Ph.D.  Candidate 
Internship:  Motorola 

Research  Project:  Queuing  theory  for  cell  phone  patients. 

Dissertation  Research:  Queuing  Theory 
Academic  Advisor:  Professor  Nikolai  Krylov 

2.  Michael  Lee,  Master’s  Graduate  Student 
Internship:  Medtronic 

Research  Project:  Comparison  of  numerical  methods  for  structural  analysis 

3.  Irina  Mitrea,  Master’s  Graduate  Student 
Internship:  3M 

Research  Project:  Model  of  electrodeposition 

4.  Martin  O’Hely,  Ph.D.  Graduate  Student 
Internship:  Lockheed  Martin 
Research  Project:  Data  fusion 

5.  Gerardo  Ortigoza,  Master’s  Graduate  Student 
Internship:  Honeywell 

Research:  Research  in  fiber  optics 


1 


6.  Svetlana  Rudnaya,  Ph.D.  Candidate 
Internship :  Ford  Motor  Company 
Research  Project :  Analysis  of  air  pollution 

Dissertation  Title  and  Research:  “Optimization  algorithm  for  designs  of  optical 
phase  masks”.  Ms.  Rudnaya  is  currently  working  on  a  problem  in  optical  phase  mask 
design  for  her  dissertation.  She  is  expected  to  graduate  in  June  1999. 

Academic  Advisor:  Professor  Fadil  Santosa 

7.  Scott  Shald,  Ph.D.  Candidate 
Internship:  Honeywell 

Dissertation  Title  and  Research:  “Problems  in  Estimation.”  His  research  involves 
the  detection  and  tracking  of  targets.  He  will  complete  his  Ph.D.  in  December  1998. 
Academic  Advisor:  Professor  Avner  Friedman. 

8.  Luis  Yunes,  Master’s  Graduate  Student 
Internship:  Banner  Engineering 

Research  Project:  Development  of  a  vision  system 

9.  Aleksandar  Zatezalo,  Ph.D.  Graduate 

Internship:  Lockheed-Martin  Tactical  Defense  Systems 
Dissertation  Title  and  Research:  “Filtering  of  Partially  Observable  Stochastic 
Processes.”  Mr.  Zatelazo,  who  completed  his  Ph.D.  in  June  1998,  focused  on 
developing  an  algorithm  for  detection  and  tracking  of  targets  in  a  cluttered 
environment.  His  research  with  Lockheed  Martin  was  on  detection  of  multi-targets. 
Academic  Advisor:  Professor  Nikolai  Krylov 


ACCOMPLISHMENTS/NEW  FINDINGS 

The  research  accomplished  by  the  Ph.D.  candidates  included: 

Scott  Shald 

Ph.D.  Expected:  1999 

Dissertation  Research  in  “tracking  problems”. 

Mr.  Shald  developed  a  scheme,  which  tracks  a  target  with  information  of  only  line  of 
sight. 

Aleksandar  Zatezalo 
Ph.D.  Awarded:  June  1998 

Dissertation  Title:  “Filtering  of  Partially  Observable  Stochastic  Processes.” 

Mr.  Zatazalo  developed  an  algorithm,  which  determines  whether,  in  the  presence  of 
noise,  there  is  a  target  in  view. 


PERSONNEL  SUPPORTED 

Faculty  partially  supported 
Avner  Friedman,  P.I. 
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Graduate  Students  supported 
Pavel  Belik,  (Ph.D.  expected  1999) 

Yi-Ju  Chao,  (Ph.D.  expected  1999) 

Michael  Lee,  (M.S.  expected  2000) 

Irina  Mitrea,  (M.S.  expected  1999) 

Martin  O’Hely,  (Ph.D.  expected  1999) 
Gerardo  Ortigoza,  (M.S.  expected  1999) 
Svetlana  Rudnaya,  (Ph.D.  expected  1999) 
Scott  Shald,  (Ph.D.  expected  1999) 

Luis  Yunes,  (M.S.  expected  1999) 
Aleksandar  Zatezalo,  (Ph.D.  received  1998) 


PUBLICATIONS 

1 .  Shald,  Scott  (Ph.D.  Candidate).  “The  Continuous  Kalman  Filter  as  the  Limit  of  the 
Discrete  Kalman  Filter,”  Stochastic  Analysis  and  Applications,  accepted. 


INTERACTIONS/TRANSITIONS  (OUTREACH  ACTIVITIES) 

During  the  1 997-98  academic  year,  Dr.  Avner  Friedman,  Director  of  the  MCIM  and  Dr. 
Fadil  Santosa,  Associate  Director  of  the  MCIM,  visited  many  companies  including 
Boeing,  Schlumberger-Doll  Research,  Lucent  Technologies,  IBM,  Kodak,  in  addition  to 
local  companies  such  as  Guidant  Computing  Devices  International,  Deluxe  Corporation, 
3M,  Honeywell,  Loram  Maintenance  of  Way,  Medtronic,  and  Lockheed-Martin.  The 
visits  helped  in  generating  interest  in  our  industrial  mathematics  graduate  studies 
program  and  research  collaborations. 

Participation/presentations  at  meetings,  conferences 

1.  Aleksandar  Zatezalo  (Ph.D.  graduate).  Second  International  Conference  on  Non- 
Linear  Problems  in  Aviation  and  Aerospace,  Orlando,  Florida,  April  29-May  1,  1998 

2.  Svetlana  Rudnaya  (Ph.D.  Candidate),  Visit  to  Ford  Research  Laboratories,  Dearborn, 
Michigan,  May  1998 

3.  Professor  Fadil  Santosa,  Presentation/Workshop,  Department  of  Mathematics,  Utah 
State  University,  June  3-5, 1998 

4.  Professor  Fadil  Santosa,  Presentation  at  the  4th  International  Conference  on  Wave 
Propagation, Golden,  Colorado,  June  12-14,  1998 

5.  Professor  Fadil  Santosa,  Presentation  at  the  National  Institute  for  Science  Education 
Conference  and  the  Graduate  Studies  Forum,  Washington,  D.C.,  June  29-30,  1998 
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6.  Professor  Fadil  Santosa,  Industrial  Mathematics  Presentations  at  Boeing  Corporation 
and  University  of  Washington  (Applied  Mathematics  Department),  Seattle, 
Washington,  August  25-28,  1998 

7.  Professor  Fadil  Santosa,  Presentation  at  the  2nd  Conference  on  Minorities  and 
Applied  Mathematicians  with  Connections  to  Industry  and  National  Laboratories, 
University  of  Califomia-Berkeley,  September  17-19,  1998 

8.  YiJu  Chao  (Ph.D.  Candidate),  Allerton  Conference  on  Communication,  Control,  and 
Computing,  University  of  Illinois  at  Urbana-Champaign,  September  23-25,  1998 


NEW  DISCOVERIES,  INVENTIONS,  OR  PATENT  DISCLOSURES 

None. 


HONORS/AWARDS 

1.  YiJu  Chao  received  a  Master’s  of  Science  degree  in  Mathematics  (emphasis  in 
Industrial  and  Applied  Mathematics)  in  March  1998.  The  title  of  her  thesis  is 
Mathematical  Approaches  to  Predictive  Health  Monitoring  for  Heart  Failure 
Patients.  Currently,  Ms.  Chao  is  pursuing  a  Ph.D.  in  Applied  Mathematics  at  the 
University  of  Minnesota  and  expects  to  graduate  June  1999. 

2.  Jennifer  Treadway  received  a  Master’s  of  Science  degree  in  Mathematics  (emphasis 
in  Industrial  and  Applied  Mathematics)  in  June  1998.  The  title  of  her  thesis  is 
Computer  Assisted  3-D  Reconstruction  from  Images.  Ms.  Treadway  currently  works 
for  Evans  and-Sutherland,  Salt  Lake  City,  Utah. 

3.  Yan  Xin  received  a  Master’s  of  Science  degree  in  Mathematics  (emphasis  in 
Industrial  and  Applied  Mathematics)  in  June  1998.  The  title  of  his  thesis  is  An 
Analysis  of  Pacemaker  Wire  Leads.  Mr.  Xin  is  currently  employed  with  LORAM 
Maintenance  of  Way. 

4.  Aleksandar  Zatezalo  received  a  Doctor  of  Philosophy  in  Mathematics  (emphasis  in 
Industrial  Mathematics)  in  June  1998.  The  title  of  his  dissertation  is  Filtering  of 
Partially  Observable  Stochastic  Processes.  Dr.  Zatezalo  currently  holds  a  post¬ 
doctoral  position  in  the  Institute  for  Mathematics  and  Its  Applications,  University  of 
Minnesota. 
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Freeman  Center  for  International  Economic  Policy 
Final  Report  to  AFOSR 


1 .  Improving  Data  Use  for  Economic  Adjustment  -  Lee  Munnich 

This  subproject  aimed  to  establish  a  major  source  of  quantitative  information  and 
techniques  to  serve  state  and  local  policymakers  engaged  in  the  process  of  economic 
reorientation  and  development. 

The  Database  for  Economic  Adjustment  work  plan  included  four  major  tasks  or  focus 
areas: 


•  identification,  collection,  and  analysis  of  information,  data  and  analytical  methods 
related  to  industry  analysis 

•  development  of  a  framework  for  industry  analysis 

•  development  and  testing  of  a  delivery  mechanism 

•  evaluation  of  the  project 

Identification,  collection,  and  analysis  of  information,  data,  and  analytical  methods 
related  to  industry  analysis.  The  project  team  conducted  a  review  of  data  sources, 
analytical  techniques,  and  frameworks  for  analysis  that  may  be  of  use  to  communities 
interested  in  understanding  their  industries  and  developing  appropriate  related  policies 
and  programs.  The  project  team  conducted  meetings  with  its  technical  and  faculty 
advisory  committees  to  further  hone  the  scope  of  the  project,  identify  and  assess  various 
data  sources  and  analytical  techniques,  and  address  specific  questions  about  the  project’s 
target  audience  and  how  to  reach  it. 

The  project  team  conducted  a  telephone  survey  of  economic  development  researchers 
from  35  states  to  identify  data  and  techniques  states  and  local  areas  use  in  analyzing  their 
industries,  measure  demand  for  a  widely  accessible  framework  for  industry  analysis  that 
includes  access  to  data,  and  determine  the  best  way  to  reach  local  and  state  economic 
development  practitioners.  In  addition  to  receiving  useful  information  on  data  and 
techniques,  the  project  team  also  learned  that  local  economic  development  practitioners 
are  in  great  need  of  an  easy-to-use  framework  for  understanding  their  industries  and 
easier  access  to  data  and  that  a  World  Wide  Web  site  would  be  the  best  way  to  reach  the 
largest  number  of  people. 

The  project  team  identified  and  studied  three  areas  that  have  used  industry-based 
strategies  to  direct  their  economic  development  efforts  and  would  serve  as  interesting  and 
instructive  examples  to  other  states  and  local  areas.  The  project  team  developed  case 
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studies  that  highlighted  these  three  areas:  Minnesota  and  two  of  its  regions,  Oregon,  and 
Cleveland,  Ohio. 


During  the  first  year  of  the  project,  a  subcontractor,  the  Northeast-Midwest  Institute 
produced  a  draft  design  of  a  labor  market  information  system  that  would  1)  consolidate 
labor  market  information  functions  from  many  parts  of  the  Department  of  Labor  into  the 
Bureau  of  Labor  Statistics;  2)  expand  the  Bureau’s  line  of  products  to  include  state  and 
local  data;  3)  provide  for  a  larger  role  for  states  in  labor  market  information 
decisionmaking;  and  4)  streamline  the  Department  of  Labor’s  funding  process  for  labor 
market  information. 

After  producing  the  initial  draft  design,  Paula  Duggan  of  the  Institute’s  staff  met  several 
times  with  policymakers  to  discuss  the  current  operation  of  the  labor  market  statistical 
system  and  how  the  draft  design  would  address  concerns  with  the  system.  Institute  staff 
also  prepared  memos  in  answer  to  particular  issues,  worked  to  refine  the  proposed  design, 
and  performed  comprehensive  analysis  of  current  labor  market  information  proposals. 

Development  of  a  framework  for  industry  analysis.  Based  on  its  research,  meetings, 
and  interviews,  the  project  team  developed  a  framework  that  state  and  local  governments 
can  use  to  analyze  their  industries  and  identify  those  that  are  of  greatest  importance  to  the 
area’s  economy. 

Development  and  testing  of  a  delivery  mechanism.  After  completing  the  research  and 
consultations  described  above,  the  project  team  decided  that  a  World  Wide  Web  site 
would  be  the  best  delivery  mechanism  to  use  since  it  would  reach  the  greatest  number  of 
people,  is  low-cost,  can  be  updated  easily,  and  could  link  communities  with  other  useful 
data  and  information  sources. 

An  Economic  Development  website  for  state  and  local  economic  development 
professionals  was  completed,  tested,  and  opened  to  public  use.  The  website  address  is 
http://www.hhh.umn.edu/Centers/SLP/edweb.  The  purpose  of  the  website  is  to  facilitate  better 
use  of  available  on-line  data  sources  and  to  improve  economic  development  policy 
making.  The  website  includes  (1)  explanations  of  how  to  apply  specific  analysis 
techniques  to  understand  regional  and  local  industries,  (2)  sample  interpretation  and 
policy  implications  of  analytical  techniques,  (3)  employment  and  data  sources  on  the 
Internet,  (4)  evaluation  of  the  relative  merits  of  each  data  site,  (5)  the  State  and  Local 
Policy  Program’s  (SLPP)  ten  Emerging  Principles  for  Economic  Development  with  24 
case  study  examples  of  how  to  apply  these  principles,  and  (6)  economic  development  and 
data  links  by  state. 

The  website  includes  the  methodology  for  conducting  industry  analysis  using  shift-share 
analysis  or  location  quotients,  links  to  data  sources  to  conduct  this  analysis,  and 
qualitative  techniques  for  understanding  industry  clusters.  The  project  team  conducted  a 
beta  test  with  forty  graduate  students  during  a  course  taught  by  Professor  Ragui  Assaad 
of  the  Humphrey  Institute.  The  beta  test  indicated  that  the  website  was  a  useful  tool  for 


2 


individuals  with  moderate  quantitative  and  computer  skills  and  provided  an  efficient  way 
of  accessing  the  necessary  data.  The  website  was  introduced  at  the  Humphrey  Institute’s 
“Innovations  in  Economic  Development”  conference  in  April  1997  and  has  also  been 
promoted  through  list  serves  and  at  other  conferences  attended  by  economic  development 
professionals.  As  more  state  and  local  governments  join  the  World  Wide  Web,  we  expect 
increasing  use  of  this  site. 

During  the  summer  of  1997  the  web  master  for  the  site  received  feedback  from  a  number 
of  users,  all  positive  or  recommending  minor  improvements.  The  information  included 
on  the  website  has  also  been  compiled  as  a  hard  copy  report  in  the  event  that  a  potential 
user  does  not  have  access  to  the  Internet  or  prefers  to  work  with  a  printed  report.  This 
report,  of  course,  does  not  include  the  data  links  and  links  to  state  economic  development 
sites  that  are  available  through  the  Internet. 

Evaluation  of  the  project.  Interviews  with  state  economic  development  professionals, 
meetings  with  the  technical  and  faculty  advisory  committees,  and  consultations  with 
other  economic  development  professionals  throughout  the  process  provided  useful 
assessments  and  ideas  for  revising  and  improving  the  project. 

To  supplement  and  expand  upon  issues  directly  related  to  applying  the  industry  based 
approach  to  economic  development,  the  State  and  Local  Policy  Program  sponsored  a 
national  conference  at  the  Humphrey  Institute  of  the  University  of  Minnesota  on  April 
1 1-12  of  1997.  This  conference  was  co-sponsored  by  the  Humphrey  Institute  and  the 
Denver-based  Center  for  the  New  West.  In  addition  to  support  from  this  grant,  the 
conference  and  proceedings  were  funded  with  grants  from  the  U.S.  Economic 
Development  Administration,  Northern  States  Power,  Advantage  Minnesota,  and  by 
participant  fees.  "This  conference  convened  170  policy  makers,  economic  development 
professionals  and  business  leaders  from  24  states  to  exchange  cutting-edge  techniques  for 
promoting  local  and  regional  economic  development  in  the  rapidly  evolving,  knowledge- 
based  economy. 

The  conference  included  a  preconference  dialogue  on  the  use  of  economic  data  for 
industry  analysis  and  economic  adjustment  and  conference  presentations  on  resources 
and  effective  strategies  for  using  economic  data  for  industry  analysis  and  program 
evaluation.  The  conference  participants  were  encouraged  to  review  and  provide  feedback 
on  the  Economic  Development  Website.  The  proceedings  of  the  conference  were 
transcribed  and  reviewed  by  speakers  and  were  professionally  edited.  The  report  is 
available  through  the  website  as  well  as  in  printed  form. 

In  May  1 997,  SLPP  convened  over  40  Minnesota  economic  development  professionals 
for  a  roundtable  discussion  regarding  an  industry  cluster  approach  to  economic 
development.  Economic  development  policies  targeted  to  clusters  of  industries  can 
improve  program  efficiency,  promote  collaboration,  and  increase  understanding  of 
growth  dynamics  with  regional  economies.  SLPP  pioneered  an  industry  cluster  analysis 
technique  through  research  in  the  Twin  Cities  and  southeastern  Minnesota.  The 
roundtable  offered  an  opportunity  for  economic  development  professionals  to  learn  more 
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about  this  emerging  approach  and  to  discuss  possible  implications  for  School-to-Work 
and  Welfare-to-Work  initiatives. 


2.  Assembling  a  Data  Base  Linking  Foreign  and  Domestic  Sectors  -  John  S. 
Chipman 

This  subproject  attacked  the  problem  of  differing  data  sets  produced  by  the  United  States 
government  for  production  and  for  foreign  trade.  The  work  focussed  on  constructing  a 
data  base  of  U.S.  imports  and  exports,  as  well  as  import  and  export  indices,  according  to 
a  classification  system  that  is  consistent  with  those  employed  in  domestic  data  such  as  the 
producer  price  index  and  index  of  industrial  production. 

The  Bureau  of  the  Census  of  the  Department  of  Commerce  has  been  making  import  and 
export  data  available  in  machine-readable  form  only  since  1989.  This  is  in  the  form  of 
CD-ROMs,  one  for  each  month.  The  CD-ROMs  contain  the  monthly  data  on  values  and 
quantities,  although  in  the  case  of  some  categories,  which  -even  at  the  finest  level  of 
classification  -contain  aggregates  (an  example  is  “rabbits  and  hares”),  no  quantity  data 
are  provided.  Even  in  the  case  of  categories  for  which  this  is  not  a  problem,  quantity  data 
are  available  for  some  shipments  but  not  all  -  for  reasons  that  remain  to  be  established. 
Professor  Chipman  was  interested  in  the  quantity  data  in  order  to  construct  a  series  on 
“volumes”  -  a  quantity  index  in  which  quantities  are  weighted  by  unit  values  in  the  base 
year.  The  quotient  of  values  by  volumes  then  furnishes  unit  values  for  the  categories. 
Since  the  import  and  export  price-index  data  furnished  by  the  Bureau  of  Labor  Statistics 
cover  only  fourth  months  per  year,  the  aim  is  to  use  the  monthly  unit- value  data  as  a 
related  series  in  order  to  interpolate  the  BLS  series  into  a  monthly  series.  Since  the 
publicly  available-data  are  available  only  since  1989,  they  could  not  be  profitably  used  in 
economic  research  unless  made  available  on  a  monthly  basis. 

The  monthly  CD-ROMs  contain  the  detailed  data  to  the  “Harmonized  System”  and  also 
the  conversion  keys  from  this  system  to  both  SIC  and  SITC  (finest  categories).  To 
estimate  volumes,  it  was  assumed  that  categories  for  which  quantity  data  are  available  are 
representative  of  those  for  which  the  quantity  data  are  unavailable;  thus  values  and 
volumes,  hence  also  unit  values,  will  be  generated  on  a  monthly  basis.  These  data  were 
also  used  to  generate  weights  for  the  BLS  price  indices,  making  it  possible  to  obtain 
monthly  price-index  data  classified  by  both  SIC  and  SITC. 

This  work  was  automated  using  the  latest  version  of  Lahey’s  Fortran  90  software.  A 
high-capacity  Zip  drive  was  also  purchased.  It  enabled  archiving  of  the  data  as  the 
project  proceeded. 

Professor  Chipman  discovered  that  the  CD-ROMs  issued  by  the  Bureau  of  the  Census 
contain  a  complete  concordance  table  allotting  each  ten-digit  Harmonized  System 
category  to  the  appropriate  finest  category  of  the  SITC,  SIC,  and  End  Use  classification 
systems.  The  Census  Bureau  also  provided  custom  made  CD-ROMs  covering  imports 
and  exports  data  not  otherwise  available. 
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It  became  clear  that  the  Census  Bureau  data  are  characterized  by  many  outliers  and 
“structural  changes”  as  described  in  the  paper,  “Computation  of  Approximate  Monthly 
Price  and  Quantity  Series  for  U.S.  Imports  and  Exports  According  to  Four  Classification 
Systems,  January  1990  -  March  1996”  (with  Hwikwon  Ham).  The  structural  changes 
result  from  an  error  in  the  method  of  producing  the  data;  the  outliers  result  from  errors  in 
the  reporting  of  the  data  by  the  primary  sources  and/or  in  entering  the  data  into  the  CD- 
ROMs.  This  research  therefore  turned  in  the  direction  of  developing  methodologies  for 
handling  outliers  and  structural  changes  in  these  data.  The  standard  statistical  literature 
on  robust  estimation  was  very  helpful,  but  does  not  cover  these  types  of  economic  time 
series,  and  had  to  be  further  developed.  The  other  challenging  problems  resulted  from 
the  fact  that  the  BLS  price-index  data  had  ceased  being  issued  in  terms  of  the  SIC 
classification,  whereas  this  was  the  crucial  classification  system  needed  to  integrate 
international  with  domestic  data.  The  more  complete  SITC  data  set  was  used  to  test 
various  methods  of  extrapolation  using  the  unit- values  series  as  related  series;  having 
found  a  satisfactory  method,  the  BLS’s  SIC  series  was  updated.  The  main  approach  of 
the  Kravis-Lipsey  school  of  thought  has  been  to  disparage  and  avoid  the  use  of  unit-value 
data  because  of  their  unreliability;  the  approach  of  this  project  was  instead  to  recognize 
that  there  is  some  valid  information  contained  in  unit-value  data,  but  it  must  be  carefully 
filtered  out. 

In  the  third  year  of  the  project,  Professor  Chipman  used  detailed  10-digit  data  to  produce 
a  set  of  monthly  data  on  U.S.  import  and  export  prices  classified  according  to  SITC,  SIC, 
and  other  classification  systems.  As  a  benchmark,  he  used  the  BLS  price-index  system. 
For  the  first  part  of  this  period  BLS  data  are  available  only  quarterly.  They  are  available 
according  to  the  SIC  in  the  early  part  of  the  period,  but  for  some  reason  have  been 
discontinued;  recent  data  have  included  data  classified  according  to  the  “Harmonized 
System”  (HS)  and  the  End  Use  classification  (EU).  Only  the  SITC  were  available  for  the 
whole  period.  Thus,  Professor  Chipman  and  his  research  assistant,  Hwikwon  Ham,  used 
the  SITC  data  to  test  their  method.  The  CD-ROMs  allowed  one  to  employ  any  four  of 
the  above-mentioned  classification  systems. 

It  is  well  recognized  by  the  most  expert  and  experienced  statisticians  working  in  the  area 
of  robust  data  analysis  (e.g.,  Tukey,  Huber,  etc.)  that  -  at  least  in  the  present  state  of  the 
art  -  one  cannot  avoid  using  subjective  graphical  methods.  The  best  one  can  do  is  to 
minimize  the  subjective  element.  Professor  Chipman  found  in  the  course  of  this  research 
that  two  types  of  outliers  must  be  sharply  distinguished:  (1)  sudden  one-period  departures 
from  “normal”  followed  in  the  next  period  by  a  return  to  “normal”;  and  (2)  sudden  once- 
for-all  changes  in  level.  He  interprets  the  latter  as  resulting  from  a  change  in  the  methods 
used  to  classify  commodities  when  new  products  are  introduced;  for  example,  some  new 
commodities  have  to  be  excluded  until  the  classification  system  (whether  HS,  SITC,  or 
other)  is  revised  to  encompass  them.  In  the  latter  case,  if  we  take  first  differences  of  the 
(logarithms  of  the)  data,  then  a  structural  change  appears  as  an  outlier  in  the  first 
differences.  When  this  outlier  is  removed  and  the  data  are  transformed  back  by 
summation,  the  structural  change,  of  course,  disappears.  This  method  was  very 
successful  in  correcting  the  unit-value  data  to  conform  to  the  price-index  data.  This 
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phenomenon  has  not  been  considered  in  any  of  the  standard  statistical  treatments  of 
outliers.  One  problem  with  this  method,  however,  is  that  if  it  is  used  when  genuine 
outliers  (as  opposed  to  structural  changes)  are  present,  it  creates  a  downward  bias  in  the 
adjusted  smoothed  data.  To  correct  for  this,  one  needs  to  use  logarithms  of  the  raw  data, 
rather  than  differences  in  these  logarithms.  Thus,  the  method  used  must  depend  to  some 
extent  on  a  subjective  evaluation  of  the  nature  of  the  singularity  in  the  data. 

Papers  resulting  from  grant  funding: 

“Computation  of  Approximative  Monthly  Price  and  Quantity  Series  for  U.S.  Imports  and 
Exports  According  to  Four  Classification  Systems,  January  1990  -  March  1996,” 
preliminary  version,  1  September  1996  (with  Hwikwon  Ham). 

“Extracting  Reliable  Information  from  Monthly  Unit  Values:  A  Problem  in  Robust 
Smoothing.” 


3.  General  Equilibrium  Modeling  of  U.S.  Adjustment  and  the  Global  Economy  - 
Timothy  J.  Kehoe 

This  subproject  refined  general  equilibrium  modeling  techniques  to  explore  the  impact  of 
changing  foreign  trade  patterns  on  specific  industries  and  regions. 

The  research  dealt  with  two  overlapping  topics: 

•  analyzing^onstructing,  and  calibrating  dynamic  general  equilibrium  models  to 
study  policies  related  to  trade  and  capital  flows. 

•  studying  the  causes  of  the  1994-95  financial  crisis  that  started  in  Mexico  and 
subsequently  spilled  over  into  much  of  the  developing  world. 

In  connection  with  the  first  topic,  Professor  Kehoe  constructed  a  calibrated  dynamic 
general  equilibrium  model  of  North  American  trade  and  capital  flows.  Preliminary 
results  indicate  that  large  capital  inflows,  an  appreciation  of  the  real  exchange  rate  and  a 
decline  in  the  domestic  savings  rate  are  exactly  what  should  be  expected  when  a 
relatively  poor,  relatively  young,  country  like  Mexico  opens  to  foreign  investment. 

In  connection  with  the  second  topic,  Professor  Kehoe  with  Caroline  Betts  of  the 
University  of  Southern  California  conducted  research  on  the  relationship  between 
fragility  in  the  domestic  banking  sector  and  currency  crises.  Evidence  suggested 
important  links  not  just  in  Mexico  in  1 994,  but  in  both  developing  and  developed 
countries  throughout  the  world  over  the  past  twenty  years.  A  dynamic  general 
equilibrium  model  was  constructed  to  study  this  relationship.  Their  empirical  work  also 
showed  that  fluctuations  in  the  U.S.-Mexico  real  exchange  rate  has  been  largely  driven 
by  changes  in  the  relative  prices  of  non-traded  goods  as  predicted  by  this  model. 
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Professor  Kehoe  also  worked  with  Gonzalo  Fernandez  de  Cordoba  of  the  Universidad  de 
Salamanca  to  use  this  model  to  explain  the  capital  flows  and  exchange  rate  movements 
that  followed  Spain’s  integration  in  the  European  Community  in  1986. 

Also  in  connection  with  the  second  topic,  Professor  Kehoe  extended  his  joint  work  with 
Harold  Cole  of  the  Federal  Reserve  Bank  of  Minneapolis.  This  work  modeled  the 
Mexican  government’s  inability  to  roll  over  its  short-term  debt  in  late  December  1994 
and  January  1995  as  a  self-fulfilling  crisis:  the  fears  on  the  part  of  investors  that  Mexico 
would  default  on  its  debt  created  conditions  in  which  a  default  seemed  inevitable.  They 
later  investigated  the  possibility  of  modeling  Mexico’s  devaluation  as  another  self- 
fulfilling  crisis. 

Professor  Kehoe  was  awarded  a  three  year,  $178,454  grant  from  the  National  Science 
Foundation  (“Modeling  International  Capital  Flows  and  Real  Exchange  Rate 
Movements,”  SBR- 96 18370)  to  continue  work  on  this  project. 

Papers  resulting  from  grant  funding: 

“Can  Debt  Crisis  Be  Prevented?”  in  Edward  Altman,  Richard  Levich,  and  Jianping  Mei, 
editors,  The  Future  of  Emerging  Market  Capital  Flows.  Richard  D.  Irwin,  1997. 

“Capital  Flows  and  North  American  Economic  Inegration,”  in  Richard  E.  Baldwin  and 

Joseph  Francois,  editors,  Dynamic  Issues  in  Applied  Commercial  Policy  Analysis. 
Cambridge  University  Press,  1997. 

Comments  on  “Are  Currency  Crises  Self-Fulfilling?”  by  Paul  Krugman,  in  Ben  S. 

Bemanke-and  Julio  J.  Rotembert,  editors,  NBER  Macroeconomics  Annual  1996. 
MIT  Press. 

“A  Dynamic  Model  of  Capital  Flows  and  Real  Exchange  Rates  in  North  America.” 
“Self-Fulfilling  Debt  Crises,”  with  Harold  L.  Cole. 

“A  Self-Fulfilling  Model  of  Mexico’s  1994-95  Debt  Crisis,”  with  Harold  L.  Cole,  Journal 
of  International  Economics.  1997. 

“Social  Accounting  Matrices  and  Applied  General  Equilibrium  Models,”  in  Ian  Begg  and 
Brian  Henry,  editors,  Applied  Economics  and  Public  Policy.  Cambridge 
University  Press,  1 997. 

Presentations  at  seminars,  conferences,  or  meetings: 

“Can  Debt  Crises  Be  Prevented?”  Conference  on  the  Future  of  Emerging  Market  Capital 
Flows,  NYU,  May  1996. 
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“Capital  Flows  and  North  American  Economic  Integration,”  National  Institute  for 
Economic  and  Social  Research,  London,  December  1995;  XX  Simposio  del 
Analisis  Economico,  Barcelona,  December  1995;  CEPR  Conference  on  Dynamic 
Trade  Policy  Analysis,  Geneva,  January  1996. 

Discussion  of  Paul  Krugman,  “Are  Currency  Crises  Self-Fulfilling?”  NBER  Macro 
Annual  Conference,  Cambridge,  Massachusetts,  March  1996. 

“A  Dynamic  Model  of  Capital  Flows  and  Real  Exchange  Rates  in  North  America,” 

Society  for  Economic  Dynamics  and  Control  Meetings,  Mexico  City,  July  1996; 
Workshop  on  Dynamic  Macroeconomics,  Universidad  de  Vigo,  July  1996. 

“The  Financial  Crisis  in  Mexico,”  Lakehead  University,  November  1995;  El  Collegio 
d’Economistes  de  Catalunya,  April  1996;  University  of  California  Riverside, 

April  1996;  Universidad  autonoma  de  Barcelona,  May  1996;  Fundacion  Caixa 
Galicia,  La  Coruna,  July  1 996. 

“Self-Fulfilling  Debt  Crises,”  University  of  Southern  California,  November  1995; 

Federal  Reserve  Bank  of  Minneapolis,  December  1995;  conference  on 
Speculative  Attacks  in  the  Global  Economy,  University  of  Maryland,  December 
1995;  North  American  Economics  and  Finance  Association  Meetings,  San 
Francisco,  January  1996;  Princeton  University,  February  1996;  Instituto 
Autonomo  Tecnologico  de  Mexico,  March  1996;  Universidad  de  Pais  Vasco,  May 
1996. 

“A  Self-Fulfilling  Model  of  Mexico’s  1994-95  Debt  Crisis,”  UCLA,  April  1996; 

Universityjof  Southern  California,  April  1996;  Universidad  de  Alicante,  April 
1996;  Universidad  Pompeu  Fabra,  May  1996;  Universidad  Carlos  III,  May  1996; 
Conference  on  Growth  and  Business  Cycles,  Santiago,  Spain,  July  1996;  NBER 
Summer  Institute,  Cambridge,  Massachusetts,  July  1996;  Conference  on  Policy 
Rules  and  Tequila  Lessons,  Buenos  Aires,  August  1996. 

“Social  Accounting  Matrices  and  Applied  General  Equilibrium  Models,”  Fiftieth 

Anniversary  of  the  Department  of  Applied  Economics  Conference,  University  of 
Cambridge,  December  1995. 

“The  Discipline  of  Applied  General  Equilibrium”  2eme  Colloque  Theories  de  la 

Macroeconomie,  Catholic  University  of  Louvain,  May  1997,  (Plenary  Address). 

“A  Dynamic  Model  of  Capital  Flows  and  Real  Exchange  Rates  in  North  America,”  Latin 
America  and  Caribbean  Economics  Association  Meetings,  Mexico  City,  October, 
1996,  UCLA,  January  1997;  Universidad  Pompeu  Fabra,  April  1997;  Federal 
Reserve  Bank  of  Atlanta,  October  1997;  Duke  University,  November  1997. 

“The  Financial  Crisis  in  Mexico,”  Roundtable  on  Speculative  Attacks,  Latin  American 
and  Caribbean  Economics  Association  Meetings,  Bogota,  October  1997. 
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“Self-Fulfilling  Debt  Crises,”  XXI  Simposio  del  Analysis  Economico,  Barcelona, 
December  1997. 


4.  Personnel  Practices  and  Economic  Adjustment  -  Morris  M.  Kleiner 

This  subproject  considered  the  relationship  between  employee  involvement  practices  and 
the  productivity  of  firms  in  both  defense  and  non-defense  sectors. 

Professor  Kleiner’s  initial  effort  focused  on  gathering  data  from  manufacturing 
organizations  that  have  had  employee  involvement  programs  and  ones  that  did  not. 
Researchers  interviewed  defense-related  establishments  and  manufacturing  plants  that 
have  large  defense  manufacturing  and  ones  that  do  not  deal  with  defense  to  examine 
whether  the  rate  of  innovation  is  greater  among  defense-related  organizations.  Over 
thirty  private  sector  organizations  agreed  to  talk  about  their  employee  involvement 
programs.  The  interviews  focused  on  the  kind  of  program  that  the  organization  has 
implemented  and  the  success  of  the  program  in  enhancing  productivity. 

Given  the  use  of  a  smaller  number  of  firms,  but  more  detailed  information  about  each 
firm,  a  new  methodological  tool  called  “Qualitative  Comparative  Analysis”  was  used. 
This  approach  employs  boolean  algebra  to  estimate  the  logic  of  the  joint  usage  of 
alternative  personnel  policies  within  an  organization.  Both  qualitative  and  quantitative 
information  was  organized  in  a  format  to  estimate  models  based  on  this  approach  first 
developed  by  Charles  Ragin  in  his  prize-winning  book,  The  Comparative  Method. 

Models  based  on-this  approach  were  estimated  during  the  summer  of  1996,  when  the  data 
collection  effort  was  complete. 

Based  on  the  research  completed  in  the  first  year  of  the  grant,  Professor  Kleiner  presented 
papers  at  the  University  of  Cambridge,  England;  INSEE,  France;  MIT,  USA;  and  the 
University  of  Minnesota. 

During  the  second  year  of  the  grant,  Professor  Kleiner  and  the  research  assistants  funded 
from  this  project  visited  35  plants  that  have  high,  low,  and  medium  levels  of  employee 
involvement.  They  compared  defense-related  industries  to  those  in  the  civilian  sector 
regarding  their  employee  involvement  programs.  In  addition,  Professor  Kleiner  and  his 
associates  developed  a  questionnaire  that  addressed  the  role  of  other  personnel  policies 
such  as  compensation  policies  within  the  employment  relationship.  The  personnel 
information  was  then  linked  to  the  LRD,  a  data  system  developed  by  the  Census  Bureau 
specifically  to  examine  changes  in  U.S.  manufacturing.  They  also  analyzed  a  new  data 
set  from  the  Society  for  Human  Resources  Management  that  contained  information  on 
employee  involvement  programs  for  over  350  publicly  traded  organizations.  This  larger 
data  set  contains  information  on  both  defense  and  nondefense  companies.  The  personnel 
policies  for  each  of  these  organizations  was  then  linked  to  productivity  and  profitability 
information  available  through  COMPUSTAT  data  on  US  firms.  Papers  using  these  data 
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sets  appeared  in  publications  at  the  National  Bureau  of  Economic  Research  and  at  a 
session  at  the  American  Economic  Association  at  its  annual  meeting  in  January,  1997. 

The  major  findings  were  that  there  were  no  major  differences  between  defense-related 
industries  relative  to  nondefense  establishments  regarding  the  impact  of  changes  in 
personnel  policies  on  economic  performance.  This  finding  is  robust  using  either  case 
studies  over  time  or  the  use  of  cross  sectional  analysis  using  publicly-traded  companies  at 
a  point  in  time. 

In  the  third  year  of  the  grant  Professor  Kleiner  produced  three  additional  papers.  In  each 
case  funding  was  used  to  gather  data  from  individual  manufacturing  establishments  and 
provide  linkages  to  information  on  economic  performance.  For  the  most  part  the  data 
from  the  defense-related  manufacturing  establishments  were  gathered  by  visits  to  the 
plants  and  interviews  with  the  persons  in  these  organizations.  The  presentations  and  a 
brief  summary  of  the  main  findings  are  noted  below. 

A  paper  entitled  “The  Anatomy  and  Effects  of  Employee  Involvement”  was  presented  at 
the  July  1997  meeting  of  the  “Productivity  Section”  of  the  National  Bureau  of  Economic 
Research  in  Cambridge,  Massachusetts.  This  study  used  support  form  the  grant  to  collect 
data  from  the  Society  for  Human  Resource  Management,  which  included  firm 
profitability  for  the  defense  and  nondefense  sectors.  The  results  using  multivariate 
maximum  likelihood  techniques  found  that  ther  is  clear  hierarchy  of  employee 
involvement  programs  that  are  used  within  firms.  However,  having  higher  levels  of 
employee  involvement  among  firms  is  not  significantly  related  to  higher  levels  of 
productivity. 

A  second  paper  that  was  given  at  the  American  Economic  Association  meeting  during 
January  1998  examined  the  role  of  employee  involvement  on  changes  in  manufacturing 
productivity  for  both  defense  and  nondefense-related  manufacturing  plants.  Among  the 
unique  aspects  of  this  study  is  the  use  of  matched  plants  by  product  line.  The  method 
examines  plants  that  had  employee  involvement  to  those  that  did  not.  The  results  are 
inconclusive  regarding  the  impact  of  employee  involvement  on  manufacturing 
productivity  over  time. 

The  third  paper  was  presented  at  the  American  Compensation  Association  research 
conference  during  March  1998.  This  paper  examined  the  impact  of  a  firm  with  defense- 
related  operations  changing  its  method  of  compensation  from  piece-rate  to  time-rate 
methods  of  pay.  The  study  attempts  to  examine  the  impact  of  the  change  in  the  pay 
incentive  system  on  the  performance  of  the  plant  that  switched  its  method  of  pay 
overtime  and  also  relative  to  those  that  did  not.  The  results  show  that  productivity 
dropped  but  that  savings  in  other  areas  such  as  worker  compensation  insurance  and 
reductions  in  inventory  resulted  in  no  effects  on  plant  level  measures  of  profitabili 
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5.  Information  and  High-Technology  in  the  Design  of  U.S.  Foreign  Economic 
Policy  -  Robert  T.  Kudrle 

This  subproject  explored  a  number  of  national  security  and  prosperity  issues  related  to 
information,  technology,  and  resource  control  in  an  increasingly  open  and  interdependent 
global  economy. 

During  the  first  year  of  the  grant  Professor  Kudrle  made  progress  in  several  areas  dealing 
with  information  about  technology  flows  in  and  out  of  the  United  States.  His  paper,  with 
his  University  of  Pittsburgh  colleague,  Davis  Bobrow,  “Should  the  U.S.  Spy  for 
Competitiveness?”  explores  various  policy  options  the  United  States  faces  in  dealing  with 
foreign  industrial  espionage,  with  or  without  state  support.  (A  longer  version  of  this 
paper  was  called  “International  Espionage  for  Commercial  Advantage.”) 

A  later  paper,  “Foreign  Acquisition  of  Defense-Related  U.S.  Firms:  Concentration, 
Competition,  and  Reality,”  examined  the  logical  and  evidentiary  foundation  for  some  of 
the  proposed  improvements  in  the  Exon-Florio  Amendment  to  the  Trade  Act  of  1988. 
Specifically,  the  paper  criticizes  the  adequacy  of  proposals  that  look  at  only  one  or  two 
aspects  of  a  defense-related  market  as  an  indicator  of  the  dangers  or  innocuousness  of 
foreign  acquisition.  The  paper  argues  strongly  that  there  are  no  simple  rules  assuring 
safety  in  defense  acquisition.  This  flatly  contradicts  suggestions  that  rules  be  based  on 
measures  of  industrial  concentration.  The  paper  argues  that  attention  to  the  nationality  of 
the  acquiring  firm  is  also  essential  for  sound  policy. 

In  the  second  year  of  the  grant  Professor  Kudrle  published  a  broader  view  of  the 
American  economy  in  its  global  context  in  “Three  Perspectives  on  U.S. 
Competitiveness^-  The  paper  concurs  with  Krugman’s  analysis  that  the  American 
economic  problem  is  one  of  productivity,  not  “competitiveness”  by  exploring  three 
different  meanings  of  the  latter  term  in  the  literature:  firm  competitiveness,  industry 
competitiveness,  and  national  competitiveness.  This  study  and  the  defense  acquisition 
paper  noted  earlier  suggest  that  a  focus  on  inadequate  economic  information  in  the  hands 
of  the  U.S.  government  may  not  be  the  generally  serious  problem  (at  least  relative  to 
other  states)  that  Laura  Tyson  implied  in  her  1992  book,  Who’s  Bashing  Whom:  Trade 
Conflict  in  High  Technology  Industries.  Nevertheless,  some  information  gaps  loom  for 
years  after  being  identified.  Perhaps  chief  among  them  is  the  absence  of  detailed 
information  abut  subcontractors  supplying  the  Departments  of  Defense  and  Energy. 

A  major  cluster  of  research  during  the  grant’s  second  year  turned  on  the  concept  of 
“globalization”  and  its  implications  for  U.S.  security  and  welfare.  The  paper 
“Globalization  and  the  Future  Policies  of  the  Industrial  States”  was  delivered  at  the 
annual  meeting  of  the  International  Studies  Association  in  San  Diego  in  March.  It  argues 
that  the  losses  of  national  control  of  economy  destiny  that  are  widely  claimed  by 
exponents  of  globalization  are  largely  a  mixture  of  the  premature  and  the  misunderstood. 
In  particular,  long-term  capital  is  not  anything  like  as  mobile  as  is  sometimes  claimed 
(because  it  is  confused  with  speculative  short-term  capital  movements.)  Only  when 
capital  mobility  increases  will  countries  find  it  necessary  to  moderate  their  capital 
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taxation  rates  to  avoid  losing  capital  to  other  states.  The  same  argument  applies  to  skilled 
labor.  Moreover,  when  capital  mobility  does  increase  -  probably  as  the  result  of 
increased  fixity  of  exchange  rates  -  the  amount  of  international  cooperation  necessary  to 
keep  states  from  competing  with  each  other  for  resources  should  be  quite  modest.  The 
immigration  of  unskilled  labor  probably  represents  the  greatest  current  threat  to 
prosperity  from  the  phenomena  generally  associated  with  the  term  globalization.  The 
revised  version  of  this  paper  was  presented  at  a  conference  at  Indiana  University  in 
October  of  1996.  A  related  paper  on  “The  Continuing  Uses  of  Economic  Nationalism,” 
was  presented  at  a  joint  meeting  of  the  Japan  Association  of  International  Relations  and 
the  International  Studies  Association,  held  in  Tokyo  in  September,  1996.  This  paper 
stressed  the  continuing  value  of  economic  nationalism  in  some  areas,  especially 
immigration,  while  noting  that  most  economic  nationalism  lowers  the  welfare  of  the 
state’s  citizens.  A  third  related  work  on  the  alternative  meanings  of  “globalization”  and 
their  public  policy  implications  was  presented  at  a  conference  at  the  University  of  Notre 
Dame  in  April  of  1997. 


All  this  research  was  conducted  with  a  common  theme  in  mind:  what  are  the  information 
requirements  and  policy  directions  that  maximize  U.S.  prosperity  and  security  in  a  world 
economy  of  constant  change? 

The  third  year  of  the  grant  continued  the  major  cluster  of  research  identified  as 
“globalization”  and  its  implications  for  U.S.  prosperity  and  security.  Professor  Kudrle 
presented  a  paper  on  fiscal  issues  connected  with  globalization  at  the  conference, 
“Coping  with  Globalization”  in  July  1998.  Some  argue  that  inter-state  competition 
resulting  from  globalization  has  removed  the  state’s  capacity  to  tax  its  citizens  and 
enterprises  at  levels  consonant  with  domestic  welfare.  Professor  Kudrle’ s  paper  argued 
that  there  is  little  evidence  that  this  is  the  case,  and  that  taxation  levels  in  the  United 
States  and  elsewhere  continue  to  be  governed  mainly  by  purely  domestic  processes, 
although  those  favoring  certain  policy  directions,  e.g.,  lower  rates  of  tax  on  capital,  are 
happy  to  invoke  international  considerations  to  argue  that  the  preferences  are  really 
driven  by  external  constraints. 

With  Davis  Bobrow,  Professor  Kudrle  extended  grant  activities  in  the  direction  of 
information  and  security  problems  in  regional  integration  schemes.  They  presented  the 
paper  “APEC  as  partisan  Mutual  Adjustment:  Foreign  Direct  Investment  and 
Competition  Policies”  at  the  Seventeenth  World  Congress  of  the  International  Political 
Science  Association  in  Seoul,  Korea,  August  7-21, 1997.  Much  of  the  summer  of  1997 
was  also  involved  in  the  preparation  of  another  paper  that  was  presented  in  Manzanillo, 
Mexico,  December  1 1,  1997.  This  work  carefully  examined  the  national  interest  issues 
connected  with  competition  policy  when  states  move  closer  together  in  regional 
integration  schemes. 
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Papers  resulting  from  grant  funding: 

“Three  Perspectives  on  Competitiveness:  An  Introduction  to  ‘Made  In  America,’”  The 
International  Executive,  July/August  1996. 

“Should  the  U.S.  Spy  for  Competitiveness?”  (with  Davis  B.  Bobrow),  Business  and  the 
Contemporary  World.  VIII,  3-4, 1996. 

“Economic  Nationalism  and  the  Global  Economy,”  in  Iliana  Zloch-Christy,  ed.,  Eastern 
Europe  and  the  World  Economy:  Challenges  of  Transition  and  Globalization. 
Edward  Elgar  Publishing,  1 997. 

“Immigration  Policy  and  National  Prosperity,”  Framtider  International.  July- August 
1997. 

“Foreign  Acquisition  of  Defense-Related  U.S.  Firms:  Concentration,  Competition  and 

Reality,”  (with  Davis  B.  Bobrow),  in  Douglas  Nigh  and  Douglas  Woodward,  eds., 
Bevond  “Us”  and  “Them”:  Foreign  Ownership  and  U.S.  Competitiveness. 
Quorum  Press,  1998. 

“Competition  Policy  and  Regional  Integration:  Compatibility  and  Conflict,”  (with  Davis 
B.  Bobrow)  Journal  of  International  Political  Economy.  March  1998. 

“Globalization  and  the  Politics  of  the  Future,”  in  Jeffrey  Hart  and  Aseem  Prakash,  eds., 
Globalization  and  Governance.  Routledge,  forthcoming. 

“APEC  as  Partisan  Mutual  Adjustment”  (with  Davis  B.  Bobrow)  in  Mary  Ann  Tetreault 
and  Kenneth  Thomas,  eds.,  Regionalism  and  Global  Political  Economy.  Lynne 
Rienner  Publishers,  forthcoming. 

“International  Espionage  for  Commercial  Advantage,”  (with  Davis  B.  Bobrow)  in  Stuart 
S.  Nagel,  ed.  Studies  in  Public  Policy  Analysis  and  Management.  St.  Martin’s 
Press,  forthcoming. 

“Three  Meanings  of  Globalization  and  Their  Implications  for  Governance,”  in  Global 
Governance  and  Enforcement:  Issues  and  Strategies.  Raimo  Vayrynen,  ed., 
Rowman  and  Littlefield,  forthcoming. 

“Aruba  and  the  Challenge  of  Globalization,”  Proceedings  of  the  Third  Aruba  Economic 
Summit.  Aruba:  Ministry  of  Economic  Affairs,  forthcoming. 

“Does  Globalization  Sap  the  Fiscal  Power  of  the  State?”  in  Aseem  Prakash  and  Jeffrey 
Hart,  eds.  Coping  With  Globalization,  manuscript  in  preparation. 
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“Investment  and  Competition  Policies  in  Regional  Perspective  (with  Davis  B. 

Bobrow), presented  at  the  International  Studies  Association  and  the  Pan-European 
International  Relations 


A  comparison  of  what  we  intended  to  accomplish  with  grant  Rinding  and  what 
was  actually  produced  gives  us  much  satisfaction.  In  each  of  the  subprojects, 
more  was  delivered  than  was  promised,  and  we  believed  that  the  funding  provided 
was  a  sound  national  investment. 
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1  Introduction 

In  the  research  program  supported  by  AF/F49620-94- 1-0461  we  have  carried  out  an  extensive 
program  in  image  processing,  vision,  and  control.  Our  methodologies  have  been  applied  to 
certain  benchmark  problems  involving  visual  tracking. 

We  have  been  working  as  well  in  distributed  parameter  control  as  part  of  the  sponsored 
research.  Recently  a  monograph  [44]  which  extensively  describes  our  skew  Toeplitz  approach 
to  distributed  H°°  design  and  analysis  has  been  published.  We  have  also  explored  problems  in 
/i-synthesis  and  analysis.  Our  results  work  directly  for  systems,  not  just  finite  matrices.  This 
allows  us  to  study  broad  classes  of  structured  perturbations  using  this  tool.  We  have  also  been 
continuing  our  work  for  a  rigorous  /z-synthesis  procedure  based  on  structured  interpolation. 

We  have  applied  our  methods  to  the  key  area  of  visual  tracking  which  may  be  employed 
for  a  number  of  problems  in  robotics,  manufacturing,  As  alluded  to  above,  we  have  developed 
robust  control  algorithms  valid  for  general  classes  of  distributed  parameter  and  nonlinear 
systems  based  on  interpolation  and  operator  theoretic  methods.  In  this  project,  we  have 
been  explicitly  combining  our  robust  control  techniques  with  our  novel  approach  to  image 
processing  in  order  to  develop  novel  visual  tracking  algorithms. 

In  active  vision,  we  have  been  conducting  research  into  advanced  algorithms  in  image 
processing  and  computer  vision  for  a  variety  of  uses:  image  smoothing  and  enhancement, 
image  segmentation,  morphology,  denoising  algorithms,  shape  recognition,  edge  detection, 
optical  flow,  shape-from-shading,  and  deformable  contours  (“snakes”).  Our  techniques  have 
already  been  applied  to  medical  imaging,  and  have  been  used  to  define  a  novel  affine  invariant 
scale-space.  Our  ideas  are  motivated  by  certain  types  of  geometric  invariant  flows  rooted  in 
the  mathematical  theory  of  curve  and  surface  evolution.  There  are  now  available  powerful 
numerical  algorithms  based  on  Hamilton-Jacobi  type  equations  and  the  associated  theory 
of  viscosity  solutions  for  the  computer  implementation  of  this  methodology.  In  addition  to 
systems  and  control,  our  work  in  vision  can  also  has  had  an  impact  on  ATR.  This  is  based  on 
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the  nonlinear  filters  we  have  developed  for  image  enhancement  that  are  natural  corollaries  of 
our  work.  These  filters  can  smooth  while  preserving  such  key  features  as  edges  and  ridges. 

2  Structured  Perturbations 

We  have  investigated  a  number  of  the  key  properties  of  the  structured  singular  value  which 
was  introduced  into  robust  control  by  John  Doyle  and  Michael  Safonov  ([34],  [85])  to  handle 
problems  involving  structured  perturbations  which  includes  both  H°°  and  the  multivariable 
gain  margin  as  special  cases.  In  particular,  we  have  introduced  a  new  lifting  or  ampliation 
method  for  the  study  of  robustness  under  a  variety  of  structured  perturbation  models.  We 
will  now  briefly  elucidate  some  of  the  key  results  in  this  area.  Details  about  this  methodology 
can  be  found  in  our  papers  [19,  17,  18,  25,  20]. 

2.1  Preliminary  Results  on  Structured  Singular  Values 

In  this  section,  we  will  introduce  some  notation  and  elementary  results  about  the  structured 
singular  value  which  we  will  need  to  explain  our  ampliation  method. 

Let  A  be  a  linear  operator  on  a  Hilbert  space  £,  and  let  A  be  an  algebra  of  operators  on 
£.  The  structured  singular  value  of  A  (relative  to  A)  is  the  number 

PA  (A)  =  l/inf{||V||  :  X  €  A,  -1  G  o{AX)}  . 

(This  quantity  was  defined  in  [34,  85]  under  a  more  restrictive  context.)  Recall  that  in  system 
analysis,  the  structured  singular  value  gives  a  measure  of  robust  stability  with  respect  to 
certain  perturbation  measures.  Unfortunately,  pa{A)  is  very  difficult  to  calculate,  and  in 
practice  an  upper  bound  for  it  is  used.  This  upper  bound  is  defined  by 

~  pa(A)  :=  inf{||V>Ur1||  :XeA',X  invertible}, 

where  A'  is  the  commutant  of  the  algebra  A. 

In  [23,  20],  we  formulated  a  lifting  technique  for  the  study  of  the  structured  singular 
value.  The  basic  idea  is  that  Pa(A)  can  be  shown  to  be  equal  to  the  structured  singular 
value  of  an  operator  on  a  bigger  Hilbert  space.  (In  [23]  this  was  done  for  finite  dimensional 
Hilbert  spaces,  and  then  in  [20]  this  was  extended  to  the  infinite  dimensional  case.)  The 
problem  with  these  results  is  that  the  size  of  the  ampliation  necessary  to  get  pa(A)  equal 
to  a  structured  singular  value,  was  equal  to  the  dimension  of  the  underlying  Hilbert  space. 
Hence  in  the  infinite  dimensional  case  we  needed  an  infinite  ampliation.  We  should  note,  that 
one  important  result  that  came  out  of  the  work  in  [23]  was  that  for  the  first  time  the  upper 
bound  p  was  shown  to  be  continuous.  Since  this  is  the  quantity  actually  used  in  p  analysis 
and  synthesis,  this  result  is  certainly  of  importance. 

In  our  most  recent  work  [19],  we  have  shown  that  in  fact,  one  can  always  get  by  with  a 
finite  ampliation.  For  the  block  diagonal  algebras  of  interest  in  robust  control,  the  ampliation 
only  depends  on  the  number  of  blocks  of  the  given  perturbation  structure.  We  will  moreover, 
give  a  new  criterion  below  showing  when  Pa{A)  =  Pa(A),  that  is,  when  no  lifting  is  necessary 
and  so  p&  (A)  gives  a  nonconservative  measure  of  robustness.  This  is  then  used  to  derive  an 
elegant  result  of  Megretski  and  Shamma  [71,  94]  on  Toeplitz  operators. 
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We  will  denote  by  £(£)  the  algebra  of  all  bounded  linear  operators  on  the  (complex, 
separable)  Hilbert  space  E.  Fix  an  operator  A  £  £(£)  and  a  subalgebra  A  C  £(£).  Observe 
that  A  C  A"  and  A'"  =  {A")'  =  A '  so  that  we  have  the  inequalities 

I^a(A)  <  /iA»(A),  /*a(A)  =  £a»(A)  . 

In  our  study  we  will  need  further  singular  values  which  we  now  define.  For  n  £  {1,2,...,  00} 
we  denote  by  E the  orthogonal  sum  of  n  copies  of  E ,  and  by  T the  orthogonal  sum  of  n 
copies  of  T  £  £(£).  Operators  on  E^  can  be  represented  a s  n  x  n  matrices  of  operators  in 
£(£),  and  is  represented  by  a  diagonal  matrix,  with  diagonal  entries  equal  to  T. 

Denote  by  An  the  algebra  of  all  operators  on  £ ^  whose  matrix  entries  belong  to  A,  and 
observe  that  (An)"  =  (A")„,  and  (An)'  =  (A')(n>  =  { T :  T  £  A'}.  Therefore  we  will 
denote  these  algebras  by  A"  and  A'n,  respectively. 

We  can  now  formulate  our  lifting  result  from  [19],  relating  /^a(A)  and  Pa(A).  For  finite 
dimensional  E,  a  lifting  result  of  this  type  was  first  proven  in  [23].  The  result  was  then 
generalized  to  the  infinite  dimensional  case  in  [20].  In  these  previous  works,  the  lifting  or 
ampliation  of  the  operator  A  and  perturbation  structure  A  depends  on  the  dimension  of  £. 
Thus  if  £  is  infinite  dimensional,  we  get  an  infinite  lifting.  In  Theorem  1  stated  below,  we  only 
have  to  lift  up  to  the  dimension  of  A'  which  in  the  cases  of  interest  in  the  control  applications 
of  this  theory  only  depends  on  the  number  of  blocks  of  the  given  perturbation  structure.  The 
proof  of  the  theorem  makes  use  of  some  work  that  we  did  on  the  relative  numerical  range  and 
the  continuity  of  the  spectrum  on  closed  similarity  orbits  in  [17, 18]  which  we  believe  also  has 
independent  interest. 

Theorem  1  Assume  that  A'  is  a  *-algebra  of  finite  dimension  n.  Then 

£a  (A)  =  »A>M(n)) 

for  every  A  £  £(£)t~ 


Remark.  In  the  cases  of  interest  in  control, 

A"  =  A, 

and  so  one  has  from  Theorem  1  that 


/*a„(A)  =  /*a  (A). 


2.2  Conditions  for  n  =  fi 

We  will  discuss  some  the  conditions  from  [19]  when  /i  =  fi  without  any  need  for  lifting  or 
ampliation.  In  such  cases,  fi  gives  a  nonconservative  measure  of  robustness  relative  to  the 
given  perturbation  structure.  In  the  finite  dimensional  case,  there  have  been  some  results 
of  this  kind,  the  most  famous  of  which  is  that  of  Doyle  [34],  who  showed  that  no  lifting  is 
necessary  for  perturbation  structures  with  three  or  fewer  blocks. 

First  of  all,  call  critical  any  Ao  G  Oa'(A)  satisfying 

limsup  ||(7  —  eX)A0(I  —  €X)_1||  >  ||Ao||,  VX  £  A', 
eto 

Then  we  have 
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Lemma  1  If  A0  is  a  critical  operator  in  0&>(A),  then  it  enjoys  the  following  property  ( O ): 

0  G  Wq (H/loll2 A'  -  AqXAo),  X  G  A', 

where  Q  =  ||Ao||2/  —  AqAo ■ 

The  next  lemma  is  the  key  step  in  adapting  the  proof  of  Theorem  1  in  order  to  show  that 

pA(A)  =  pa{A) 

in  several  interesting  cases. 

Lemma  2  Let  Ao  be  an  operator  on  S  which  satisfies  the  essential  version  of  property  ( G ), 
property  (0°),  namely 

OeW%(\\A0\\‘1X-A'0XAo),XeA,t 

where  Q  =  ||Ao||2/-  AqA0-  Then  there  exists  a  sequence  C  £,  ||/ijt||  =  1,  k  =  1,2, 

such  that 

Qhk  ->  0  strongly  and  ((||A0||2A  -  AlXAQ)hk,  H)  -»•  0, 

for  all  X  G  A'. 

We  can  now  state  another  key  result: 

Theorem  2  If  there  exists  a  critical  operator  A0  satisfying  property  G°  in  the  closed  A' -orbit 
of  A,  then 

Ha"(A)  =  Pa{A). 


Remark.  Under  the  hypotheses  of  Theorem  2,  when  A"  —  A  (which  happens  in  all  cases  of 
interest  in  control),  we  have  that 

Pa(A)  =  pa{A). 

Let  L(A'AA')  denote  the  linear  space  generated  by 

A'  A  A'  —  {XAY  :  X,Y  G  A'}. 

Obviously  L(A'AA')  is  finite  dimensional,  and  therefore  closed.  Hence  Ga>(A)  C  L(A'AA>). 

Corollary  1  If  for  every  B  G  L(A'AA'),  B  /  0,  the  norm  of  B  is  not  attained  (that  is, 
there  is  no  h  £%  such  that  ||B/i||  =  ||£||||/i||  ^  0),  then 

PA"  [A)  =  pa{A). 
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3  Nonlinear  Control 


In  our  work  in  nonlinear  robust  control,  and  especially  the  extension  of  H°°  to  the  nonlinear 
framework,  we  have  discovered  a  number  of  intriguing  problems  which  do  not  occur  in  the 
linear  case.  Indeed,  besides  the  theoretical  and  practical  questions  involved  in  finding  an 
implementable  design  methodology,  it  is  interesting  to  note  that  certain  associated  problems 
of  causality  have  arisen  in  working  in  this  area,  which  we  would  like  to  briefly  sktch.  In  fact, 
as  a  result  of  this  effort  we  have  been  able  to  put  an  explicit  causality  constraint  in  commutant 
lifting  theory  for  the  first  time  [40,  42,  48,  16]. 

There  have  been  several  attempts  to  extend  dilation  theoretic  techniques  to  nonlinear 
input/output  operators,  especially  those  which  admit  a  Volterra  series  expansion  (see  [41] 
and  the  references  therein).  In  several  of  these  approaches,  one  is  reduced  to  applying  the 
classical  (linear)  commutant  lifting  theorem  to  an  H2- space  defined  on  some  Dn  (where  D 
denotes  the  unit  disc).  Now  when  one  applies  the  classical  result  to  Dn  (n  >  2),  even  though 
time-invariance  is  preserved  (that  is,  commutation  with  the  appropriate  shift),  causality  may 
be  lost.  Indeed,  for  analytic  functions  on  the  disc  D ,  time-invariance  (that  is,  commutation 
with  the  shift)  implies  causality.  For  analytic  functions  on  the  n— disc  (n  >  1),  this  is  not 
necessarily  the  case.  For  dynamical  system  control  design  and  for  any  physical  application, 
this  is  of  course  a  major  drawback.  Hence  for  a  dilation  result  in  H2(Dn)  we  need  to  include 
the  causality  constraint  explicitly  in  the  set-up  of  the  dilation  problem.  We  will  discuss  a  way 
of  doing  this  now  based  on  [48,  41,  42]. 

3.1  Causality  in  Nonlinear  Systems 

Causality  basically  means  that  for  a  given  input/output  system  the  past  output  is  independent 
of  the  future  inputs.  This  may  be  given  precise  mathematical  formulation  in  terms  of  a  family 
of  projections  which  we  shall  now  do.  We  follow  treatment  of  [48]. 

Let  S  denote  an  isometry  on  a  Hilbert  space  Q,  and  let  T  denote  a  contraction  on  a 
Hilbert  space  U.  Let  P,o,  j  >  1  denote  a  sequence  of  orthogonal  projections  in  Q  satisfying 
the  following  conditions: 


Pro  <  -P20  <  •  •  •  (1) 

Pjo  <  I-SjS*j  j  —  1,2,...  (2) 

Pj+i, 0*5(7  —  Pjo)  =  0  j  =  l,2 .  (3) 

Next  U  :  1C  -*  K  will  denote  a  minimal  isometric  dilation  of  the  given  contraction  T 
defined  on  %.  Let  B  :  Q  — >■  H  intertwine  S  with  U,  that  is, 

UB  =  BS.  (4) 

Note  that  this  latter  condition  implies  U^B  =  BS J  (j  >  1),  hence  UW*iBS* S**  =  BS^B*\ 
and  so 

(7  -  UJU*j)B  =  (/  -  UjU*j)B{I  -  SjS*j)  j  =  1,2,....  (5) 

We  now  make  the  following  key  definition: 
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Definition.  An  operator  B  satisfying  (4)  is  called  (Pio,  -P20,  •  •  •)- causal  (and  if  the  sequence 
{PjfdJLi  is  fixed,  causal)  if 

(/  _  UjU*j)B  =  (7  -  UjU*j)BPj0  j  >  1,  (6) 

or  equivalently, 

(7  -  Pjo)B*  =  (7  -  Pj0)B*UjU*j  j  >  1.  CO 

Note  that  B  is  always  (7  -  SS*,I  -  S2S*2,...)-c ausal.  In  what  follows  the  sequence 
p10,  P20,  •  •  •  will  be  fixed  and  causality  will  always  be  defined  relative  to  this  sequence. 

Next  we  let  A  :  Q  -4  71  be  an  operator  intertwining  S  and  T,  that  is,  AS  =  T A.  Then 
an  intertwining  lifting  (or  dilation )  of  A  is  an  operator  B  \Q  K  such  that  BS  —  UB ,  and 
PB  =  A  where  P  :  K,  -4  ?7  denotes  orthogonal  projection. 

We  define 

(A)  :=  inf{||B||  :  B  is  a  causal  intertwining  dilation  of  A}, 

and 

H{A)  :=  min{M  >  0  :  ||A||  <  M,  ||(7  -  Pj0)A*/i||  <  M\\T^h\\,  h  G  K  j  >  1}. 

We  can  then  show  that 
A*(A)  <  t'ooM)- 

We  will  also  need  a  functional  which  lies  between  /^(A)  and  t,00(A).  To  this  aim,  we  call 
a  sequence  of  operators  Tj  :  Qj  :=  (7  —  Poj)G  — t  %,  j  =  0,2, . . .  a  resolution  of  A  if 

To  =  A 

IjlGj+i  =  Trj+i  Vy  >  0, 

r,  =  ri+1s|^  Vi>0. 

(Note  that  we  take  P00  :=  0,  so  that  G0  =  G)  We  now  define 

/2(A)  := 

min {M  >  0  :  ||A||  <  A7  and  there  exists  a  resolution  of  A,  Tj,  with  ||Fj||  <  M,  Vj  >  0}. 
Then,  we  have  the  following  results  from  [48]: 

Theorem  3  (Causal  Commutant  Lifting  Theorem)  Notation  as  above.  Then 

v co  (A)  =  /1(A). 
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3.2  Iterated  Nonlinear  Optimization 

We  will  now  consider,  how  the  above  ideas,  lead  to  a  an  optimization  procedure  for  nonlinear 
systems  which  generalizes  H°°  in  a  completely  natural  manner.  For  convenience,  we  will 
only  treat  SISO  systems  here.  Let  us  call  an  analytic  input/output  operator  <j>  :  H2  — »• 
H 2  admissible  if  is  is  causal,  time-invariant,  majorizable,  and  ^>(0)  =  0.  Denote  the  set  of 
admissible  operators  by  Ca.  In  what  follows  below,  we  assume  P,W  E  Ca,  and  that  W  admits 
an  admissible  inverse. 

We  consider  the  (one  block)  problem  of  finding 

ps  :=  inf  sup  ||[(J  +  P  o  C)-1  o  W]v\\,  (9) 

c  IHI<« 

where  we  take  the  infimum  over  all  stabilizing  controllers.  (In  what  follows,  we  let  ||  ||  denote 
the  2-norm  ||  | (2  on  H2  as  well  as  the  associated  operator  norm.  The  context  will  make  the 
meaning  clear.)  Thus  we  are  looking  at  a  worst  case  disturbance  attenuation  problem  where 
the  energy  of  the  signals  v  is  required  to  be  bounded  by  some  pre-specified  level  6.  (In  the 
linear  case  of  course  since  everything  scales,  we  can  always  without  loss  of  generality  take 
5  =  1.  For  nonlinear  systems,  we  must  specify  the  energy  bound  a  priori.)  Then  one  sees 
that  (9)  is  equivalent  to  the  problem  of  finding  the  problem  of  finding 

ps  =  inf  sup  ||(W  —  Po  g)u||.  (10) 

96CoIMI<* 

The  iterated  causal  commutant  lifting  procedure  gives  an  approach  for  approximating  a 
solution  to  such  a  problem.  Briefly,  the  idea  is  that  we  write 

W  =  Wi  +W2  +  •  •  • , 

P  =  Pi  +  P2  +  --, 

9  =  9i  +  92  d - > 

where  Wj,  Pj,  qj  are  homogeneous  polynomials  of  degree  j.  Notice  that 

ps  =  6  inf  ||W1-P19i||  +  0(52),  (11) 

where  the  latter  norm  is  the  operator  norm  (i.e.,  H°°  norm).  From  the  classical  commutant 
lifting  theorem  we  can  find  an  optimal  (linear,  causal,  time-invariant)  q\i0pt  E  H°°  such  that 

PS  =  S\\W1-P1qhopt\\  +  0(S2).  (12) 

Now  the  iterative  procedure  gives  a  way  of  finding  higher  order  corrections  to  this  lin¬ 
earization.  Let  us  illustrate  this  now  with  the  second  order  correction.  Indeed,  having  fixed 
the  linear  part  9i,opt  of  q  in  (10),  we  note  that 

W(v)  -  P(q(v))  -  (Wt  -  Piqhopt)(v)  = 

w2{v)  -  P2{qi,opt{v))  -  Pi92(u)  +  higher  order  terms. 

Regarding  W2 ,  P2, 92  as  linear  operators  on  H2  ®  H2  =  H2(D2,  C)  as  above,  we  see  that 

sup  \\(W-Po  q)(v)  -  {Wx  -  Piqi,0PM\  <  52||W2  -  P192II  +  0(53), 

IMI<* 
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where  the  “weight”  W2  is  given  by 

W 2  :=  W2  —  P2{<h,opt  ®  9i ,opt)- 

Using  the  control  version  of  the  causal  commutant  lifting  theorem  (see  [41]  and  Section  4.1 
below),  we  can  now  construct  an  optimal  admissible  q2, opt,  and  so  on.  This  is  our  iterated 
optimization  procedure. 

In  short,  instead  of  simply  designing  a  linear  compensator  for  a  linearization  of  the  given 
nonlinear  system,  this  methodology  allows  one  to  explicitly  take  into  account  the  higher  order 
terms  of  the  nonlinear  plant,  and  therefore  increase  the  ball  of  operation  for  the  nonlinear 
controller.  Moreover,  if  the  linear  part  of  the  plant  is  rational,  our  iterative  procedure  may 
be  reduced  to  a  series  of  finite  dimensional  matrix  computations.  (See  [41,  45]  for  discussions 
of  rationality  in  the  nonlinear  framework.) 


4  Curve  Evolution  in  Vision 

In  the  past  few  years,  we  have  become  very  interested  in  visual  tracking  and  the  general  area 
of  the  use  of  visual  information  in  a  feedback  loop.  This  is  a  central  area  in  which  the  robust 
control  methods  developed  over  the  past  fifteen  years  could  have  a  major  impact.  In  order 
to  work  on  visual  tracking,  we  have  had  to  learn  some  of  the  key  techniques  from  image 
processing  and  computer  vision,  which  has  led  in  turn  to  a  new  research  direction.  Indeed, 
we  have  been  using  geometric  invariant  flows  for  various  problems  in  active  vision.  These 
flows  themselves  are  very  much  motivated  by  ideas  in  optimal  control;  see  [65].  We  will  now 
discuss  some  of  the  key  ideas  in  curve  evolution.  These  will  be  applied  to  certain  problems 
in  controlled  active  vision  as  described  below. 


4.1  Curve  Evolution 


A  geometric  set  or  shape  can  be  defined  by  its  boundary.  In  the  case  of  bounded  planar 
shapes  for  example,  this  boundary  consists  of  closed  planar  curves.  We  will  only  deal  with 
closed  planar  curves,  keeping  in  mind  that  these  curves  are  boundaries  of  planar  shapes. 

A  curve  may  be  regarded  as  a  trajectory  of  a  point  moving  in  the  plane.  Formally,  we 
define  a  curve  C()  as  the  map  C(p)  :  S1  R2  (where  S 1  denotes  the  unit  circle).  C.  can 
be  written  using  Cartesian  coordinates,  i.e.,  C{p)  =  [x{p),y{p)]T,  where  x()  and  y(-)  are 
maps  from  S1  to  R.  We  assume  that  all  of  our  mappings  are  sufficiently  smooth,  so  that 
all  the  relevant  derivatives  may  be  defined.  We  also  assume  that  our  curves  are  have  no 
self-intersections,  i.e.,  are  embedded. 

We  now  consider  plane  curves  deforming  in  time.  Let  C(p,t)  :  S1  x  [0,r)  R2  denote  a 
family  of  closed  embedded  curves,  where  t  parametrizes  the  family,  and  p  parametrizes  each 
curve.  Assume  that  this  family  evolves  according  to  the  following  equation: 


{ 


§  =  af  +  pti 
C(p,0)  =  Co(p) 


(13) 


where  Kr  is  the  inward  Euclidean  unit  normal,  T  is  the  unit  tangent,  and  a  and  /?  are  the 
tangent  and  normal  components  of  the  evolution  velocity  v,  respectively.  In  fact,  it  is  easy 
to  show  that  Img[C(p,t)]  =  Img[C(u;,  f)L  where  C(p,t )  and  C{w,t)  are  the  solutions  of 


Ct  =  af  +  0  and  Ct  =  P tf, 
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respectively.  (Here  Img[*]  denotes  the  image  of  the  given  parametrized  curve  in  R2,)  Thus  the 
tangential  component  affects  only  the  parametrization,  and  not  Img[-]  (which  is  independent 
of  the  parametrization  by  definition).  Therefore,  assuming  that  the  normal  component  (5  of  V 
(the  curve  evolution  velocity)  in  (13)  does  not  depend  on  the  curve  parametrization,  we  can 
consider  the  evolution  equation 

<i4) 


where  /?  =  V  •  N ,  i.e.,  the  projection  of  the  velocity  vector  on  the  normal  direction. 

The  evolution  (14)  was  studied  by  different  researchers  for  different  functions  j3.  This  type 
of  flow  was  introduced  into  the  theory  of  shape  in  [61,  62,  63,  64].  One  of  the  most  studied 
evolution  equations  is  obtained  for  ft  =  k,  where  k  is  the  Euclidean  curvature: 


(15) 


Equation  (15)  has  its  origins  in  physical  phenomena  [3,  53].  It  is  called  the  Euclidean 
shortening  flow ,  since  the  Euclidean  perimeter  shrinks  as  fast  as  possible  when  the  curve 
evolves  according  to  (15);  see  [53].  Gage  and  Hamilton  [51]  proved  that  a  planar  embedded 
convex  curve  converges  to  a  round  point  when  evolving  according  to  (15).  (A  round  point  is 
a  point  that,  when  the  curve  is  normalized  in  order  to  enclose  an  area  equal  to  n,  it  is  equal 
to  the  unit  disk.)  Grayson  [52]  proved  that  a  planar  embedded  non-convex  curve  converges 
to  a  convex  one,  and  from  there  to  a  round  point  from  Gage  and  Hamilton  result.  Note  that 
in  spite  of  the  local  character  of  the  evolution,  global  properties  are  obtained,  which  is  a  very 
interesting  feature  of  this  evolution.  For  other  results  related  to  the  Euclidean  shortening 
flow,  see  [3,  4,  51,  52,  53,  64,  102]. 

Next  note  that  jf.s  denotes  the  Euclidean  arc-length,  then 


kM  = 


82C 
Os 2  ’ 


Therefore,  equation  (15)  can  be  written  as 


Ct  =  Css.  (16) 

Equation  (16)  is  not  linear,  since  s  is  a  function  of  time  (the  arc-length  gives  a  time  dependent 
parametrization).  This  equation  is  also  called  the  geometric  heat  equation. 

Another  interesting  example  is  obtained  when  one  sets  (5  —  1  in  equation  (14).  This 
equation  simulates,  under  certain  conditions,  the  grassfire  flow  [27,  61,  62,  64,  92].  This 
grassfire  flow  is  also  the  morphological  scale-space  created  by  a  disk.  One  can  show  that  with 
different  selections  of  /3,  other  morphological  scale-spaces  are  obtained  [65]. 

In  [61,  62,  64],  we  have  studied  the  following  equation  in  order  to  develop  a  hierarchy  of 
shape, 

^  =  (1  +  €K)tf.  (17) 

If  e  —¥  0  in  (17),  the  grassfire  flow  is  obtained,  and  this  introduces  singularities  ( shocks )  in 
the  evolving  curve.  (The  shocks  define  the  well-known  skeleton.)  On  the  other  hand,  if  c  ->  oo, 
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equation  (17)  reduces  to  the  classical  Euclidean  curve  shortening  flow,  which  smoothes  the 
curve  [52,  93].  The  combination  of  these  two  opposite  features  gives  very  interesting  properties 
[61]. 

When  a  curve  evolves  according  to  (17),  the  evolution  of  the  curve  slope  satisfies  a  reaction- 
diffusion  equation  [95].  The  reaction  term,  which  tends  to  create  singularities,  competes 
with  the  diffusion  term  which  tends  to  smooth  the  curve.  For  each  different  value  of  e, 
a  scale-space  is  obtained  by  looking  at  the  solution  of  (17),  and  considering  the  time  t  as 
the  scale  parameter.  We  have  called  the  set  of  all  the  scale-spaces  obtained  for  all  values 
of  €,  the  reaction- diffusion  scale-space  [62].  In  particular,  we  see  that  the  much  studied 
Euclidean  shortening  flow  (equation  (15))  defines  an  Euclidean  invariant  scale-space  (the 
equation  admits  Euclidean  invariant  solutions). 

We  now  discuss  the  affine  analogue  of  the  Euclidean  shortening  flow.  (The  affine  group 
SA2  is  the  group  generated  by  unimodular  transformations  and  translations  of  R2.  Under 
certain  natural  conditions,  it  provides  a  good  approximation  to  the  full  group  of  perspective 
projective  transformations.)  Then  in  [77,  87],  we  show  that  the  simplest  non-trivial  affine 
invariant  flow  in  the  plane  is  given  by 

Ct  =  k1/^.  (18) 

The  question  now  is  what  happens  when  a  non-con  vex  curve  evolves  according  to  (18). 
The  following  result  answers  this  question  [5]: 


Theorem  4  Let  C(-,0)  :  51  — >  R2  be  a  smooth  embedded  curve  in  the  plane.  Then  there 
exists  a  family  C  :  S1  x  [0,  T)  -4  R2  satisfying 

Ct  =  , 

such  that  C(-,t)  islmooth  for  all  t  <  T,  and  moreover  there  is  a  to  <  T  such  that  for  all 
t  >  to,  C(-,t)  is  smooth  and  convex. 


Theorem  4  means  that  just  as  in  the  Euclidean  case,  a  non-convex  curve  first  becomes 
convex  when  evolving  according  to  (18).  After  this,  the  curve  converges  to  an  ellipse  from  our 
results  in  [87].  Because  of  this,  and  other  related  properties  (see  [88]),  we  can  conclude  that 
equation  (18)  is  the  affine  analogue  of  (15)  for  smooth  embedded  curves,  and  thus  is  called 
the  affine  shortening  flow.  (It  is  also  the  affine  invariant  formulation  of  the  geometric  heat 
equation.)  In  the  next  section,  we  will  use  it  to  construct  an  affine  invariant  scale-space  for 
planar  shapes. 

4.2  Affine  invariant  scale-space 

In  the  previous  section,  we  indicated  that  equation  (18)  is  the  affine  analogue  of  (15).  Of 
course,  in  addition  to  the  Euclidean  invariant  property  of  the  Euclidean  curve  shortening 
flow,  this  affine  flow  admits  also  unimodular  affine  invariant  solutions.  Due  to  this  analogy, 
and  certain  key  properties  [88],  we  can  use  the  affine  shortening  flow  for  defining  a  new  affine 
invariant  scale-space.  More  precisely,  the  representation  of  shape  obtained  from  the  evolving 
solution  of  equation  (18),  defines  the  affine  invariant  scale-space.  The  scale  parameter  is  given 
by  the  time  t 
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Note  the  curve  shrinks  when  evolving  according  to  (18).  This  curve  can  be  normalized 
in  order  to  keep  constant  area.  In  fact,  in  [89],  we  have  evolutions  that  preserve  either  area 
or  length.  The  affine  scale-space  can  therefore  be  re-defined  by  means  of  these  normalized 
dilated  curves.  Since  the  process  of  normalization  modifies  k)  and  other  geometric  properties 
just  by  constant  multiplication  [87],  all  the  key  scale-space  features  for  the  family  of  curves  C 
satisfying  (18),  also  hold  for  the  normalized  scale-space. 

When  defining  scale-spaces,  a  number  of  desired  properties  must  be  verified  [56,  66,  80, 
103].  One  of  the  most  important  properties  is  the  causality  criterion ,  which  requires  that 
no  new  features  should  be  introduced  in  the  curve  (or  image)  in  passing  from  fine  to  coarse 
scales  in  the  scale-space.  In  general,  “features”  are  related  to  the  salient  characteristics  of  the 
signal,  which  are  important  for  the  image  description,  and  are  easily  identified. 

Hummel  in  [56]  showed  that  the  causality  criterion  may  be  expressed  in  terms  of  the  Max¬ 
imum  Principle  [82].  The  Maximum  Principle  states  that  under  certain  conditions,  a  given 
function  satisfying  a  parabolic  partial  differential  equation,  attains  its  maximum  (minimum) 
on  the  boundary  [82].  (The  Maximum  Principle  is  very  frequently  used  in  the  theory  of  curve 
evolution  for  proving  important  properties.)  Hummel  showed  that  under  certain  conditions, 
when  the  Maximum  Principle  holds  for  a  given  parabolic  operator,  which  defines  the  scale- 
space,  zero-crossings  are  never  created  at  a  non-zero  scale.  (This  result  was  used  in  [80]  for 
demonstrating  the  causality  principle.)  In  our  work  [5,  88],  we  show  that  the  affine  shortening 
flow  satisfies  a  Maximum  Principle,  and  so  because  of  this  and  other  related  properties,  the 
affine  invariant  scale-space  is  an  ideal  vehicle  for  the  multi-scale  study  of  shape. 

4.3  Visual  Tracking 

Much  of  our  recent  research  in  image  processing  and  computer  vision  has  been  motivated  by 
problems  in  controlled  active  vision,  especially  visual  tracking.  We  have  already  described 
some  of  the  relevant-work  in  control  above,  and  so  we  would  like  to  consider  now  some  of  the 
key  tools  we  plan  to  employ  from  our  work  in  computer  vision  and  image  processing.  These 
include  active  contours,  optical  flow  and  stereo  disparity,  and  certain  results  from  invariant 
theory  for  invariant  object  recognition,  as  well  as  the  curve  and  surface  evolution  methodology 
sketched  above  in  Section  4.  These  methods  have  played  an  integral  part  in  our  study  of  the 
utilization  of  visual  information  in  a  feedback  loop. 

4.4  Geometric  Active  Contours 

In  this  section,  we  will  describe  a  new  paradigm  for  snakes  or  active  contours  based  on  prin¬ 
ciples  from  geometric  optimization  theory.  Active  contours  may  be  regarded  as  autonomous 
processes  which  employ  image  coherence  in  order  to  track  various  features  of  interest  over 
time.  Such  deformable  contours  have  the  ability  to  conform  to  various  object  shapes  and  mo¬ 
tions.  Snakes  have  been  utilized  for  segmentation,  edge  detection,  shape  modeling,  and  visual 
tracking.  Active  contours  have  also  been  widely  applied  for  various  applications  in  medical 
imaging.  For  example,  snakes  have  been  employed  for  the  segmentation  of  myocardial  heart 
boundaries  as  a  prerequisite  from  which  such  vital  information  such  as  ejection-fraction  ratio, 
heart  output,  and  ventricular  volume  ratio  can  be  computed. 

In  the  classical  theory  of  snakes,  one  considers  energy  minimization  methods  where  con¬ 
trolled  continuity  splines  are  allowed  to  move  under  the  influence  of  external  image  dependent 
forces,  internal  forces,  and  certain  contraints  set  by  the  user.  As  is  well-known  there  may 
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be  a  number  of  problems  associated  with  this  approach  such  as  initializations,  existence  of 
multiple  minima,  and  the  selection  of  the  elasticity  parameters.  Moreover,  natural  criteria 
for  the  splitting  and  merging  of  contours  (or  for  the  treatment  of  multiple  contours)  are  not 
readily  available  in  this  framework. 

In  [60],  we  have  proposed  a  novel  deformable  contour  model  to  successfully  solve  such 
problems,  and  which  will  become  one  of  our  key  techniques  for  tracking.  Our  method  is 
based  on  the  Euclidean  curve  shortening  evolution  (see  Section  4.1)  which  defines  the  gradient 
direction  in  which  a  given  curve  is  shrinking  as  fast  as  possible  relative  to  Euclidean  arc-length, 
and  on  the  theory  of  conformal  metrics.  Namely,  we  multiply  the  Euclidean  arc-length  by  a 
function  tailored  to  the  features  of  interest  which  we  want  to  extract,  and  then  we  compute  the 
corresponding  gradient  evolution  equations.  The  features  which  we  want  to  capture  therefore 
lie  at  the  bottom  of  a  potential  well  to  which  the  initial  contour  will  flow.  Further,  our  model 
may  be  easily  extended  to  extract  3D  contours  based  on  motion  by  mean  curvature  [60]. 

We  first  choose  a  function  <j>{x,y)  which  depends  on  the  given  image  and  is  used  as  a 
“stopping  term.”  For  example,  the  term  <j>(x,  y)  may  chosen  to  be  small  near  an  edge,  and  so 
acts  to  stop  the  evolution  when  the  contour  gets  close  to  an  edge.  One  may  take 

a - I _ ,  (19) 

1  +  ||VG,  *  ZIP’  v  ' 

where  I  is  the  (grey-scale)  image  and  G„  is  a  Gaussian  (smoothing  filter)  filter. 

Using  <j>,  we  now  define  our  geometric  snake  algorithm  based  on  ideas  from  Euclidean 
curve  shortening.  Indeed,  we  will  change  the  ordinary  Euclidean  arc-length  function  along  a 
curve  C  =  {x(p),y{p))T  with  parameter  p  given  by 

ds=  (Zp  +  yl)1/2dp, 


—  ds#  =  (x^  +  yp)l/2<f>dp. 

Then  we  want  to  compute  the  corresponding  gradient  flow  for  shortening  length  relative  to 
the  new  metric  ds^. 

Accordingly  set 

r  1 

M0-/o  WTfWv- 


Let 


-  dC  ...dc, 

T:=^V 


denote  the  unit  tangent.  Then  taking  the  first  variation  of  the  modified  length  function 
and  using  integration  by  parts  (see  [60]),  we  can  show  that  the  direction  in  which  the 
perimeter  is  shrinking  as  fast  as  possible  is  given  by 


^  =  (4>k-  (20) 

This  is  precisely  the  gradient  flow  corresponding  to  the  miminization  of  the  length  functional 
L The  level  set  version  of  this  is 


(21) 
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One  expects  that  this  evolution  should  attract  the  contour  very  quickly  to  the  feature  which 
lies  at  the  bottom  of  the  potential  well  described  by  the  gradient  flow  (21).  As  in  [30,  73],  we 
may  also  add  a  constant  inflation  term,  and  so  derive  a  model  given  by 

^  =  #l|V®||(div(jj||jj)  +  0  +  V#-V«.  (22) 

Notice  that  for  <j>  as  in  (19),  V<£  will  look  like  a  doublet  near  an  edge.  Of  course,  one  may 
choose  other  candidates  for  <f>  in  order  to  pick  out  other  features. 

We  now  have  very  fast  implementations  of  these  snake  algorithms  based  on  level  set 
methods  [79,  92].  Clearly,  the  ability  of  our  snakes  to  change  topology,  and  quickly  capture 
the  desired  features  will  make  them  an  indispensable  tool  for  our  visual  tracking  algorithms. 

We  are  also  studying  an  affine  invariant  snake  model  for  tracking.  (The  evolution  itself 
works  using  a  level  set  model  of  kWv  as  discussed  in  Section  4.1.) 


5  Concluding  Remarks 

We  have  just  outlined  some  of  the  key  findings  that  we  derived  in  our  research  under  AFOSR 
Contract  AF/F49620-94-1-0461.  Much  of  the  work  is  still  continuing.  We  will  briefly  describe 
some  of  these  projects  here. 

First  of  all  in  control,  we  are  extending  our  work  in  robust  control  from  linear  time- 
invariant  systems  to  the  time-varying  case.  Further,  we  are  considering  a  game-theoretic 
global  approach  to  nonlinear  robust  control  in  contrast  to  the  power-series  methodology  con¬ 
sidered  above. 

In  computer  vision,  we  are  developing  very  general  segmentation  algorithms  for  both  3D 
and  4D  segmentation.  These  snake-based  techniques  will  also  be  very  useful  in  image  tracking. 
We  are  also  exploring  putting  in  probabilistic  information  in  our  models,  that  is  combining 
Bayesian  statistics  with  curve  and  surface  evolution.  We  are  also  considering  new  ideas  for 
shape  representation  which  can  be  useful  for  a  number  of  applications  including  registration 
in  biomedical  imaging. 

The  final  goal  is  a  well-developed  theory  of  using  vision  in  a  feedback  loop,  which  we 
regard  as  one  of  our  major  research  challenges.  This  goal  still  faces  a  number  of  daunting 
technical  problems  before  it  can  be  realized. 
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Final  Report  to  AFOSR 

Ahmed  Tewfik,  Department  of  Electrical  and  Computer  Engineering 

1.  Research  in  Signal  Processing  Partially  Supported  by  Grant  AF/F49620-94-1-0461: 

Research  in  the  area  of  signal  processing  focused  on  two  main  topics:  watermarking/data  embedding  for  audio, 
image  and  video,  and  analysis  and  content-based  retrieval  of  visual  data.  Additional  work  was  done  in  the  areas  of 
low  power  communications  schemes  for  mobile  multimedia,  MIDI  transcriptions  of  audio  signals  and  time  scale 
modifications  of  audio  signals. 

1.A  Main  Research  Topics: 

In  this  subsection,  we  will  provide  an  introduction  to  watermarking/data  embedding  for  audio,  image  and  video, 
and  analysis  and  content-based  retrieval  of  visual  data  and  summarize  our  main  accomplishments  in  these  areas. 
Detailed  descriptions  of  the  results  that  we  have  obtained  in  these  two  areas  can  be  found  in  several  journal  and 
conference  papers  (see  Section  5  below)  as  well  as  in  the  theses  of  J.  Nam,  B.  Zhu  and  M.  Swanson.  Additional 
results  will  be  reported  in  J.  Nam’s  Ph.D.  thesis  which  is  due  to  be  completed  in  Spring  1999. 

1.A.1  Data  Embedding  and  Watermarking 

Digital  representation  has  become  the  preferred  format  for  exchanging,  manipulating,  storing  and  playing  back 
audio,  image  and  video  signals.  Digital  media  has  a  high  perceptual  quality  and  is  easily  reproduced,  transmitted, 
stored  and  edited.  These  advantages  have  opened  up  many  new  possibilities.  In  particular,  it  is  possible  to  embed 
data  (information)  within  audio,  image  and  video  files.  The  information  is  hidden  in  the  sense  that  it  is 
perceptually  and  statistically  undetectable  (c.f.  Fig.  1).  With  many  schemes,  the  hidden  information  can  still  be 
recovered  if  the  host  signal  is  compressed,  edited  or  converted  from  digital  to  analog  format  and  back. 

Data  embedding  is  also  referred  to  as  watermarking, ,  data  hiding  and  fingerprinting  in  the  literature.  It  has  many 
applications.  Foremost  is  passive  and  active  copyright  protection.  Many  of  the  inherent  advantages  of  digital 
signals  increase  problems  associated  with  copyright  enforcement.  For  this  reason,  creators  and  distributors  of 
digital  data  are  hesitanrto  provide  access  to  their  intellectual  property.  Copyright  information  can  be  hidden  in  an 
audio,  image  or  video  file  by  watermarking  the  file,  i.e.,  making  small  imperceptible  modifications  to  its  samples. 
Unlike  encryption,  watermarking  does  not  restrict  access  to  digital  media.  Further,  a  watermark  stays  with  the 
media  after  decryption.  It  is  intended  to  provide  a  solid  proof  of  ownership. 


Embedded  Info 
(e.g.,  text,  audio, 
video,  etc.) 

Fig.  1:  Block  diagram  of  data  embedding  procedure 
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Data  embedding  also  provides  a  mechanism  for  inserting  additional  information  in  a  given  signal.  This 
information  can  be  used  for: 

•  Signal  description:  by  recording  important  reference,  control  and  descriptive  information  about  the  signal, 
e.g.,  an  identifier  code,  capture  and  reproduction  information  (recording,  camera,  scanner  or  printer 
parameters),  description  of  the  signal  content,  etc., 

•  Passive  rights  management  and  protection: 

tracking  the  use  or  license  of  a  particular  clip,  e.g.,  for  pay-per-use  applications,  including  billing 
for  commercials  and  video  and  audio  broadcast,  and  internet  electronic  commerce  of  digital 
media, 

auditing  audio  or  visual  object  creation  dates,  manipulation  and  modification  history  within  a  given 
signal,  signal  creators  and  editors,  without  the  overhead  associated  with  creating  a  separate  header  or 
history  file, 

•  Data  privacy  and  active  rights  management  and  protection:  by  linking  viewing  or  playing  of  content  to  an 
individual  media  player  or  group  of  players, 

•  E-commerce:  by  linking  the  audio,  image  or  video  signal  to  a  sales  site  where  users  can  buy  the  streamed 
audio  or  video  signal  that  they  are  hearing  or  viewing, 

•  Mass  Customkation: 

Customized  advertisements  embedded  in  streamed  audio  or  video, 
multicasting, 

personalized  multimedia  delivery  including  providing  multiple  access  levels  to  a  given  signal, 

•  Fraud  and  tamper-detection  in  audio,  image  or  video, 

•  Fast  download  of  web  pages  with  rich  multimedia  content. 

A  most  interesting  example  of  the  last  application  is  personalized  video  delivery.  Personalized  video  delivery 
allows  users  to  watch  a  movie,  broadcast  to  all  viewers  over  a  single  channel ,  in  a  particular  rating  or  in  a  given 
language  of  their  choice.  In  this  case,  data  embedding  is  used  to  embed  extra  scenes  and  multilingual  tracks  in  the 
given  version  of  the  movie  that  is  broadcast.  Each  user  then  extracts  the  hidden  information  that  is  needed  to 
reconstruct  the  movie  in  the  rating  and  language  that  he  selected.  In  a  sense,  data  embedding  then  provides  some 
of  the  capability  of  DVD  in  a  broadcast  environment  with  no  extra  bandwidth  or  storage  requirements. 

Data  embedding  and  watermarking  research  builds  on  ideas  and  concepts  developed  in  cryptography, 
communications  theory,  algorithm  design  and  signal  processing.  The  data  embedding  problem  is  inherently  more 
difficult  than  any  of  the  problems  that  have  traditionally  been  addressed  in  these  fields.  All  data  embedding 
algorithms  combine  and  extend  in  a  sense  many  of  the  solutions  developed  in  these  areas. 

An  effective  data  embedding  scheme  must  at  the  minimum  be  able  to: 

•  insert  data  in  the  digital  signal  without  deteriorating  its  perceptual  quality, 

•  retrieve  the  data  without  reference  to  the  original  host  signal.  (This  is  not  a  requirement  of  watermarking 
schemes  which  may  have  access  to  the  original.) 


2 


In  many  applications,  the  data  embedding  scheme  must  also: 

•  provide  a  large  data  embedding  capacity,  i.e.,  be  able  to  hide  the  largest  number  of  bits  in  a  given  signal.  This 
is  particularly  important  for  personalized  video  delivery  or  fast  download  of  multimedia  rich  web  pages. 

•  be  robust ,  i.e.,  be  able  to  retrieve  the  data  even  when  the  host  signal  has  undergone  modifications,  such  as 
compression,  editing  or  translation  between  formats,  including  A/D  and  D/A  conversions  (e.g.,  printing, 
faxing,  copying  to  analog  cassette  or  VCR  tape). 

•  be  secure,  i.e.,  statistically  undetectable :  An  unauthorized  user  should  not  be  able  to  recover  the  hidden  data 
by  statistically  analyzing  several  media  examples  with  hidden  information.  This  should  be  true  even  if  he 
knows  the  exact  data  embedding  algorithm  used,  as  long  as  one  input  to  the  algorithm  (the  “key”)  is  not 
known.  Further,  an  unauthorized  user  should  not  be  able  to  establish  that  data  has  actually  been  hidden  in  a 
given  audio,  image  or  video  file. 

The  security  of  the  data  embedding  algorithm,  like  that  of  most  cryptographic  procedures,  depends  on  the  security 
of  the  key.  The  Serial  Copy  Management  System  developed  by  the  digital  audio  tape  (DAT)  industry  is  an 
example  of  a  poor  data  embedding  and  copy  protection  scheme.  The  location  of  the  copy  protection  status 
information  on  the  tape  is  fixed  and  known.  As  a  result,  hardware  and  software  are  widely  available  to  remove 
copyright  protection  from  a  DAT. 

Watermarking  algorithms  that  protect  copyrights  have  another  additional  stringent  requirement: 

•  they  must  provide  unambiguous  proof of  ownership. 

Although  this  requirement  is  intuitively  obvious,  almost  all  vendors  and  academic  research  groups  have  not 
recognized  it  as  such.  Multiple  users  can  insert  digital  watermarks  in  a  given  digital  signal.  Unless  the 
watermarking  scheme  can  infer  the  order  in  which  the  watermarks  have  been  added,  a  deadlock  occurs  and  it 
becomes  impossible  to  establish  who  produced  or  owns  the  signal.  Unlike  other  data  embedding  applications, 
watermarking  for  copyright  protection  can  make  use  of  the  original  digital  signal  for  watermark  detection.  The 
rightful  owner  will  typically  store  a  copy  of  the  original  and  could  make  it  available  along  with  a  key  to  a  third 
party  that  attempts  to~establish  ownership  of  a  signal.  Watermarking  schemes  that  require  the  original  data  set  for 
watermark  detection  can  also  suffer  from  deadlocks  if  the  watermark  is  not  properly  designed.  In  such  schemes,  a 
party  other  than  the  owner  may  counterfeit  a  watermark  by  “subtracting  off’  a  second  watermark  from  the 
publicly  available  data  and  claim  the  result  to  be  his  or  her  original. 

Accomplishments: 

Our  research  group  has  pioneered  several  data  embedding  techniques  with  a  number  of  unique  features: 

•  truly  imperceptible  data  embedding  in  an  audio,  image  or  video  host  signal, 

•  highest  data  embedding  capacity ,  including  the  only  demonstrated  video-in-video  application, 

•  highly  robust  hidden  data  recovery,  even  when  the  host  signal  had  been  edited,  compressed  or  undergone 
translation  between  formats  or  A/D  and  D/A  conversions  (printing,  faxing,  copying  to  analog  tape  or  VCR), 

•  true  data  privacy 

•  secure  embedded  data , 

•  deadlock  free ,  i.e.,  our  watermarking  techniques  provide  unambiguous  proof  of  ownership, 

•  true  video  data  embedding  and  watermarking, 
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Imperceptibility  and  high  capacity 

The  first  two  features  of  our  data  embedding  and  watermarking  techniques  stem  from  a  unique  data  insertion 
technique  that  exploits  the  characteristics  of  the  human  visual  and  auditory  system. 

Data  insertion  is  possible  because  a  human  ultimately  consumes  the  digital  media  The  human  hearing  and  visual 
systems  are  imperfect  detectors.  Audio  and  visual  signals  must  have  a  minimum  intensity  or  contrast  level  before  a 
human  can  detect  them.  These  minimum  levels  depend  on  the  spatial,  temporal  and  frequency  characteristics  of 
the  human  auditory  and  visual  systems.  Further,  the  human  hearing  and  visual  systems  are  characterized  by  an 
important  phenomenon  called  masking.  Masking  refers  to  the  feet  that  a  given  audio  or  visual  signal  may  become 
imperceptible  in  the  presence  of  another  signal  called  the  masker.  Most  signal  coding  techniques  exploit  the 
characteristics  of  the  human  auditoiy  and  visual  systems  directly  or  indirectly.  Likewise,  all  data  embedding 
techniques  exploit  the  characteristics  of  the  human  auditory  and  visual  systems  implicitly  or  explicitly.  In  fact,  data 
embedding  would  not  be  possible  without  the  limitations  of  the  human  visual  and  auditory  systems.  For  example, 
it  is  not  possible  to  modify  a  binary  stream  that  represents  programs  or  numbers  that  will  be  interpreted  by  a 
computer.  The  modification  would  directly  and  adversely  affect  the  output  of  the  computer. 

Almost  all  of  the  data  embedding  and  watermarking  schemes  available  on  the  market  modify  the  host  signal 
without  explicitly  computing  the  maximum  amount  of  imperceptible  modification  that  can  be  introduced  within 
each  signal  sample  or  signal  transform  domain  coefficient.  Since  the  modification  must  be  imperceptible,  portions 
of  the  video,  image  or  audio  where  modifications  are  most  easily  detectable,  e.g.,  smooth  regions  in  an  image, 
limit  the  magnitude  of  the  modification.  As  a  result,  the  data  extraction  performance  of  these  schemes  is  generally 
more  susceptible  to  signal  perturbations,  e.g.,  compression,  editing  and  noise  addition. 

In  contrast,  our  data  embedding  and  watermarking  techniques  explicitly  compute  masking  information  as  a 
function  of  space  and  time.  By  using  this  explicit  masking  information,  our  data  embedding  techniques  guarantee 
imperceptibility.  Furthermore,  they  can  evaluate  the  information  bearing  capacity  of  any  given  signal  and 
determine  how  to  embed  the  maximum  possible  amount  of  information  in  that  signal. 

Typically,  blind  tests  are  used  to  assess  the  perceptual  transparency  of  data  embedding  procedures.  In  such  tests, 
subjects  are  randomly  presented  signals  with  and  without  embedded  data  and  asked  to  determine  which  signal  has 
a  perceptually  higher  ^quality.  A  probability  of  selecting  the  signal  with  no  embedded  data  that  is  roughly  equal  to 
50  percent  is  indicative  of  perceptual  transparency.  Table  1  illustrates  the  perceptual  transparency  of  our  image 
data  embedding  technique. 


Image 

Probability  that  original  is  preferred 

Lena 

53.33% 

F16 

49% 

Houses 

49% 

Mandrill 

49.67% 

Peppers 

48% 

Table  1:  Examples  of  blind  test  results  conducted  for  our  image  data  embedding  technique. 
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Table  2  provides  information  about  the  current  capabilities  of  our  data  embedding  procedures.  The  amount  of  data 
that  can  be  embedded  in  a  given  signal  decreases  with  increased  robustness  requirements.  For  example,  the 
amount  of  data  that  can  be  embedded  and  successfully  extracted  from  a  signal  that  may  undergo  digital  to  analog 
conversion  (e.g.,  recording  on  an  analog  audio  or  video  tape,  faxing  or  printing)  is  less  than  that  which  can  be 
embedded  in  a  signal  that  is  guaranteed  to  remain  unedited  (e.g.,  a  signal  stored  in  read-only  form  in  a  data  base). 
Fig.  2  shows  an  original  512x512  gray  scale  image  and  the  same  image  with  4  Kbytes  of  data  embedded  in  it 


Host 

Signal 

Byte  rate  or  byte  density 

Audio 

2-250  bytes/sec  in  mono  CD  quality  audio  signal 

Image 

8-256  bytes  in  8  bits  gray  scale  128x128  image 

Video 

900  -  92 1 6  bytes/sec  in  320x240  8  bit  gray  scale  24  frames/sec  video 

Table  2:  Data  embedding  rates  for  our  data  embedding  approaches.  The  lower  byte  rates  correspond  to 
the  highly  robust  embedding  mode  that  can  survive  severe  compression  and  editing  including  digital  to  analog 
conversion.  The  higher  byte  rates  are  less  robust  but  will  tolerate  modest  compression  and  editing. 


(a) 


(b) 


Fig.  2:  (a)  Original  image,  (b)  Image  with  1  Kbytes  of  data  embedded. 


Robustness 

Our  data  embedding  and  watermarking  techniques  achieve  robustness  by  properly  shaping  the  data  prior  to 
insertion.  Shaping  is  based  on  explicit  masking  information.  It  ensures  that,  after  data  embedding,  the  change  in 
each  sample  of  the  host  signal  has  die  largest  possible  magnitude  while  remaining  imperceptible.  Furthermore,  the 
shaping  is  designed  to  maximize  the  invariance  of  the  embedded  data  to  the  modifications  that  can  occur  when  the 
host  signal  is  edited,  compressed,  converted  from  a  format  to  another,  printed,  scanned,  faxed  or  recorded  on 
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analog  media.  Since  detection  performance  is  proportional  to  the  energy  of  the  embedded  signal  and  the 
robustness  of  the  detector  to  changes  in  the  embedded  signal,  our  data  embedding  and  watermarking  techniques 
have  the  highest  data  extraction  performance  with  excellent  perceptual  quality. 

Tables  3-4  provide  examples  of  the  robustness  of  our  data  embedding  procedures.  We  note  also  that  we  can 
recover  up  to  1  Kbytes  of  embedded  data  from  a  512x512  gray  scale  image  that  is  compressed  at  75% 
JPEG  quality. 


192  Kb/s 

100% 

128  Kb/s 

100% 

96  Kb/s 

100% 

Table  3.  Robustness  of  the  audio  data  embedding  procedure  at  different  MPEG  coding  rates. 


Image 

Bit  recovery 

Lena 

100% 

F16 

100% 

#  of  embedded  bits 

128  in  128x128  gray  scale  image 

Table  4.  Illustration  of  the  robustness  of  the  image  data  embedding  procedure  to  printing  and 

scanning. 


Data  Privacy 

Our  data  embedding  solution  offers  robust  data  privacy.  The  host  audio,  image  or  video  signal  is  scrambled 
using  secure  cryptographic  procedures  and  the  de-scrambling  information  is  securely  embedded  in  the  host 
signal  itself.  Unlike  encryption,  this  data  privacy  solution  is  robust  to  transmission  errors  and  compression. 
Furthermore,  the  data  remains  private  as  it  is  converted  from  one  format  to  another. 

Embedded  data  security 

The  security  of  our  data  embedding  and  watermarking  techniques  results  from  using  random  insertion  and 
modification  patterns  that  cannot  be  broken  or  guessed  in  any  realistic  time.  Note  that  the  classical  maximal  length 
pseudo  noise  sequences  that  many  schemes  use  are  insecure.  These  sequences  are  generated  by  linear  feedback 
shift  registers.  The  feedback  pattern  (i.e.,  the  keys)  can  easily  be  guessed  given  a  small  number  of  output  bits. 

Unambiguous  proof  of  ownership 

Our  watermarking  techniques  provide  unambiguous  proof  of  ownership  by  using  a  dual  watermarking  scheme 
and  properly  designed  watermarking  sequences.  The  watermarks  depend  strongly  on  the  digital  signal  that  they 
are  intended  to  protect.  In  the  dual  watermarking  scheme,  a  pair  of  watermarks  is  embedded  into  the  signal.  One 
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watermark  is  detected  with  the  help  of  the  original  data  set  The  second  watermark  is  detected  without  reference  to 
the  original  data  set  That  watermark  is  intended  to  mark  the  signal  that  a  pirate  may  claim  as  his  or  her  original. 
Either  of  these  schemes  alone  could  provide  unambiguous  proof  of  ownership. 

True  video  data  embedding  and  watermarking 

Unlike  almost  all  other  approaches  available  on  the  market  which  treat  video  as  a  sequence  of  image  stills,  our 
video  data  embedding  and  watermarking  approach  takes  into  account  the  temporal  dimension  of  a  video  sequence. 
As  a  result,  this  scheme  can  hide  more  information  in  a  given  video  and  is  much  less  susceptible  to  attacks  and 
detection. 

1.A.2  Content  Based  Retrieval 

Digital  audio,  image  and  video  signals  are  playing  an  increasingly  important  role  in  the  new  multimedia  world. 
This  has  led  to  the  creation  of  large  repositories  of  audio,  image  and  video  information.  The  proliferation  of  such 
repositories  has  produced  a  major  challenge:  how  does  one  efficiently  access  the  auditory  and  visual  information 
that  is  stored  in  these  databases?  Today,  the  largest  cost  factor  associated  with  the  creation  of  multimedia 
applications  is  undoubtedly  the  cost  of  retrieving  and  editing  the  relevant  audio,  image  or  video  information  that 
resides  in  such  repositories. 

Currently,  most  audio,  image  and  video  management  systems  rely  on  textual  description  of  their  content  or 
keywords.  This  textual  data  is  referred  to  as  metadata.  The  metadata  is  organized  in  a  standard  database.  Searching 
can  then  be  done  using  standard  Boolean  textual  database  queries,  exact  or  probabilistic  matches  of  the  query  text. 
Retrieval  can  be  enhanced  by  using  a  thesaurus  and  taxonomies.  Taxonomies  consist  of  hierarchical  of  subject 
classes  that  organize  the  data  at  various  levels,  e.g.,  semantic  (politics,  fashion,  etc.),  visual  (planes,  people,  etc.). 

A  text-only  based  approach  to  the  organization  of  audio,  image  or  video  databases  has  several  drawbacks. 
First,  many  auditory  and  visual  characteristics  of  the  data  cannot  be  adequately  described  in  words,  e.g.,  a 
particular  pitch,  color  or  shape.  More  importantly,  such  an  approach  necessarily  relies  on  human 
intervention.  Human  input  is  expensive.  Further,  the  descriptions  that  human  produce  are  highly  subjective. 
As  a  result,  queries  may  produce  inaccurate  and  incomplete  results  or  may  fail  entirely. 

To  deal  with  the  subjective  nature  of  human  descriptions,  strong  standardization  efforts  are  under  way.  These 
include  the  MPEG-7  standard,  extension  of  the  Dublin  Core  to  images  and  an  initiative  of  the  Library  of  Congress 
aimed  at  standardizing  the  taxonomy  of  graphic  material. 

On  the  other  hand,  researchers  and  users  have  come  to  the  realization  that  the  full  information  content  of  audio, 
image  and  video  signals  can  only  be  exploited  if  these  signals  are  managed  and  manipulated  as  visual  or  auditory 
signals.  The  natural  approach  to  managing  and  retrieving  audio  or  visual  data  is  to  deal  with  the  auditory,  visual 
and  temporal  content  of  the  signal  such  as  pitch,  loudness,  timbre,  rhythm,  temporal  and  spatial  patterns,  colors, 
shapes,  motion,  layout  and  location  information,  etc.  Queries  based  on  such  auditory,  visual  or  temporal 
information  may  supplement  text-based  queries  and  in  many  instances  are  the  easiest  or  only  way  of  formulating  a 
given  query.  A  content-based  data  management  and  retrieval  system  is  a  system  that  relies  primarily  on  auditory, 
visual  or  temporal  information  to  manage  and  retrieve  audio,  image  and  video  data.  Such  systems  may  be  used  in 
conjunction  with  text-based  approaches  or  may  incorporate  higher  level  text-based  searching  and  indexing 
features. 

Content-based  retrieval  systems  differ  conceptually  from  traditional  text-based  retrieval  system  in  one  major 
respect.  Retrievals  are  based  on  similarity  rather  than  exact  matches.  As  a  result,  these  systems  act  more  like 
information  filters.  False  positives  may  be  retrieved  and  need  to  be  manually  discarded. 

Several  criteria  can  be  used  to  assess  the  usefulness  of  any  content-based  retrieval  system.  These  criteria  include: 

•  Visual  and  auditory  interface:  content-based  retrieval  system  use  auditory  and  visual  information 
to  formulate  a  query.  Further,  they  may  return  many  false  positives  since  they  look  for  similarity 
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rather  than  exact  matches.  User  feedback  therefore  plays  an  important  role  in  refining  the  query. 
Hence,  such  systems  need  advanced  visual,  auditory  and  temporal  interfaces. 

•  Automation:  Retrievals  cannot  be  done  by  exhaustively  looking  for  given  patterns  in  all  signals  in 
the  database.  Instead,  query  and  retrieval  operations  examine  short  form  important  auditory,  visual 
or  temporal  features  of  the  signal.  The  process  of  extracting  these  auditory  and  visual  content 
identifiers  is  known  as  indexing.  Indexing  can  be  done  manually  (requiring  human  intervention), 
semi-automatically  or  fully  automatically. 

•  Adaptability:  An  important  feature  of  a  content-based-retrieval  system  is  its  ability  to  dynamically 
adapt  its  indexing  and  retrieving  function  according  to  the  relevance  feedback  provided  by  the 
user. 

•  Query  nature  and  abstraction  level:  The  nature  of  the  queries  varies  with  the  domain  and 
application.  Queries  can  involve  bibliographic  information  (e.g.,  title),  subject  content  (e.g.,  F-16), 
structural  information  (e.g.,  shot),  action  content  (e.g.,  taking  off),  syntax  (e.g.,  open  spaces)  and 
subjective  component  (e.g.,  description  of  a  mood  or  impression).  Most  current  commercial 
content-based  query  systems  are  based  on  color  and  texture  queries.  They  have  limited  filtering 
capability  and  tend  to  retrieve  many  false  positives. 

•  Categorization:  Content-based  retrieval  systems  should  act  as  sophisticated  information  filters.  As 
such,  they  provide  machine  assisted  browsing.  Browsing  can  range  from  free  style  to  category 
guided  navigation.  In  many  instances,  the  user  does  not  know  what  he  or  she  wants  to  retrieve 
exactly.  Retrieval  proceeds  by  displaying  representative  examples  of  the  signals  in  the  database. 
Dynamic  reorganization  of  the  information  based  on  relevance  feedback  can  drastically  cut  search 
time. 

•  Scope:  Another  important  feature  of  content-based  retrieval  system  is  their  ability  to  catalog  and 
index  networked  as  well  as  local  or  server  information. 

•  Memory  requirements  (index  size):  Content-based  retrieval  systems  create  an  index  that  is 
accessed  during  the  search.  An  improperly  designed  index  may  have  a  size  equal  to  that  of  the 
database.  On  the  other  hand,  a  properly  designed  index  will  use  a  minimum  amount  of  additional 
disk  space  while  supporting  a  full  range  of  query  features. 

•  Speed  of  retrieval:  In  most  applications  of  content-based  retrieval  systems,  the  query  must  be 
answered  while  the  user  is  waiting  in  front  of  a  terminal.  Furthermore,  in  many  cases  a  successful 
retrieval  will  involve  several  iterations.  Real  time  resolution  of  the  queries  is  therefore  critical. 

•  Precision  and  recall:  The  value  of  a  content-based  retrieval  system  is  directly  proportional  to  the 
quality  of  the  information  filtering  function  that  it  provides.  That  quality  can  be  measured  in  terms 
of  the  false  positives  that  the  system  returns  as  well  as  its  ability  to  retrieve  all  the  relevant  signals 
in  the  database.  The  false  positive  issue  is  particularly  important  when  dealing  with  large 
databases. 

Content-based  retrieval  systems  will  play  a  dominant  role  in  many  applications  including: 

•  Multimedia  document  techniqueion 

•  Electronic  consumer  transaction  systems,  e.g.,  electronic  catalogs  with  multimedia  content,  on-line 
sale  of  audio,  image  and  video  content,  etc. 

•  Trademark  systems,  e.g.,  logo  search 

•  Law  enforcement 
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•  Geographical  information  systems 

•  Medical  imaging 

•  Advertisement  techniqueion 

•  Studios 

•  Broadcast 

•  Personalized  information  delivery 
Accomplishments 

We  have  developed  a  content-based  retrieval  system  with  several  novel  key  features: 

•  Fast  retrieval 

•  Low  memory  requirements 

•  Support  for  visual  query  by  example ,  sketch,  shape  or  color 

•  Highly  accurate 

•  User  friendly  visual  interface 

•  Fast  browsing  of  networked  information 

•  Accurate  video  analysis  tools 


query 

retrieval 

query 

retrieval 

(a) 

(b) 

Fig.  3  Screen  captures  illustrating  our  content-based  image  retrieval  system  and  its  query  type 
support:  a)  Query  by  example,  b)  query  by  sketching. 

Fast  retrieval  and  low  memory  requirements 

Our  content-based  image  and  video  retrieval  software  uses  a  unique  index  organization  and  compression  approach 
that  minimizes  index  size  as  well  as  query  response  time.  In  particular,  the  index  is  optimized  for  a  progressive 
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resolution  of  the  query.  Queries  are  decomposed  in  a  hierarchical  manner.  At  each  detail  level,  the  query  is 
resolved  using  binary  string  matching  procedures:  no  complex  measures  are  computed. 

Visual  queries 

Our  content-based  image  and  video  retrieval  software  supports  visual  object  based  query  by  example,  shape  or 
sketch.  To  formulate  a  query,  a  user  simply  draws  a  box  around  an  object  of  interest  in  the  image  displayed  on  the 
screen.  Alternatively,  the  user  can  sketch  an  object  The  system  then  retrieves  other  images  with  similarly  looking 
objects. 

We  are  currently  extending  the  system  to  support  color  based  queries  and  Boolean  combinations  of  queries. 

High  accuracy 

Several  factors  account  for  the  high  retrieval  performance  of  our  content-based  image  and  video  retrieval  software. 
First,  the  software  supports  object  based  example  specification.  This  ability  to  specify  more  than  just  the  color  or 
texture  of  the  object  of  interest  dramatically  reduces  false  positives.  Second,  the  software  uses  a  smart  B-spline 
based  representation  of  the  object  contour.  This  representation  provides  a  large  degree  of  invariance  to  point-of- 
view.  Therefore,  the  software  can  extract  images  that  contain  objects  viewed  from  a  different  angle  than  the  object 
in  the  query.  Third,  the  software  allows  the  user  to  combine  multiple  queries.  Combinations  of  queries  can  tighten 
the  specification  of  the  desired  object  of  interest 

Browsing: 

Our  content-based  image  retrieval  software  features  a  unique  object  based  progressive  rendering  of  images.  At  the 
lowest  resolution  level,  objects  in  an  image  are  represented  by  generic  members  of  their  class.  For  example,  a 
generic  plane  may  serve  as  a  substitute  for  a  Boeing  747.  This  unique  feature  allows  the  user  to  browse  images  on 
remote  servers  rapidly  while  getting  a  good  understanding  of  the  actual  composition  of  the  images. 

Furthermore,  the  software  automatically  organizes  images  or  videos  based  on  content.  This  enables  it  to 
provide  dynamic  category-based  browsing,  where  the  definition  of  a  given  category  and  membership  in  that 
category  are  dynamically  refined  in  response  to  user  relevance  feedback. 

Accurate  video  analysis  tools 

Our  video  content-based  retrieval  software  uses  a  unique  video  segmentation  algorithm.  The  segmentation 
algorithm  breaks  the  video  into  shots,  scenes  and  stories.  It  produces  key  frames  based  on  a  statistical 
analysis  of  the  video  shots  and  indexes  these  key  frames  as  well  as  transition  information  between  key 
frames.  The  algorithm  also  classifies  the  genre  of  each  shot,  e.g.,  violence,  romance,  etc.,  for  accurate 
indexing  and  retrieval. 

l.B  Other  Areas  of  Research 

We  have  made  several  advances  in  three  related  areas:  low  power  mobile  multimedia  communications,  MIDI 
transcription  of  music  and  time-scale  modifications  of  audio  signals.  The  results  that  we  have  obtained  in  these 
areas  are  described  in  several  journal  and  conference  papers  (see  Section  5  below)  as  well  as  in  the  theses  of  N. 
Sieger  and  those  of  K.  Hamdy  and  T.-S.  Lan.  These  last  two  theses  will  be  completed  in  Spring  1999. 


l.B.l  Low  Power  Mobile  Multimedia  Communications 

We  were  the  first  group  to  formulate  and  solve  the  problem  of  minimizing  the  total  energy  consumed  by  a  mobile 
multimedia  terminal.  Our  approach  consisted  in  formulating  an  optimization  problem  that  involves  the  energy 
consumed  by  the  coding,  error  correction  and  modulation  operations  involved  in  such  systems  subject  to  a  quality 
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constraint  We  solved  the  problem  both  for  static  image  transmission  as  well  as  video  transmission.  For  video 
transmission,  we  focused  on  mode  switching  in  the  new  H.263  low  bit  rate  video  coding  format. 

1.B.2  MIDI  Transcription  of  Music 

We  have  developed  a  suite  of  algorithms  for  sound  segmentation  and  pitch  detection  of  individual  notes.  Our  pitch 
extraction  algorithms  rely  on  fitting  the  observed  data  with  entries  from  a  dictionary  made  of  harmonic  spectra 
from  samples  of  music  sounds.  Our  experiments  indicate  the  approach  can  reliably  deal  with  monophonic  sounds 
and  some  polyphonic  sounds. 

1. B 3  Time-Scale  Modifications  of  Audio  Signals 

We  have  developed  a  new  approach  to  time  scale  modifications  of  audio  signals  based  on  the  detection  of  hyper- 
peridocities  in  the  signal  and  the  decomposition  of  basic  signal  units  into  harmonic  compoenents,  attacks  and 
residuals.  Our  approach  uses  a  different  phylosophy  than  traditional  audio  time  scaling  algorithms.  It  provides  a 
marked  improvement  in  the  perceptual  quality  of  the  time-scaled  audio  pieces. 

2.  Personnel  Partially  Supported  by  Grant  AF/F49620-94-1-0461: 

a  Senior 

1.  Prof.  A.  H.  Tewfik 
b.  Research  Assistants 

1.  Mitchell  Swanson 

2.  Bin  Zhu 

3.  Khalid  Hamdy 

4.  Jeho  Nam 

5.  Keesook  Han 

6.  Tse  Hua  Lan 

7.  Nicholas  Sieger 

3.  Thesis  Completed  with  Partial  Support  from  Grant  AF/F49620-94-1-0461 

A.  M.Sc.  Students: 

1 .  Jeho  Nam,  Thesis  on  “Combined  Audio  and  Visual  Streams  Analysis  for  Video  Sequence 
Segmentation,”  December  1996. 

2.  Nicolas  Sieger,  Report  on  “AudioCoding  for  MIDI”,  October  1997. 


B.  Ph.D.  Students: 

1.  Mitchell  Swanson,  Thesis  on  “Issues  in  Multimedia  Databases:  Coding  for  Content-Based  Image  Retrieval 
and  Digital  Copyright  Protection”,  March  1997. 


11 


2.  Bin  Zhu,  Thesis  on  “Coding  and  Data  Hiding  for  Multimedia”,  July  1997. 

4.  Honors  and  Awards  of  Prof.  A.  H.  Tewfik  in  the  Period  1994-1998 

1.  Distinguished  Lecturer  of  the  IEEE,  Signal  Processing  Society,  199 7. 

2.  Fellow  IEEE,  1996. 

3.  Invited  Principal  Lecturer  at  the  1995  IEEE  EMBS  Summer  School,  Siena,  Italy,  July  1995. 

4.  Invited  Tutorial  Instructor  at  the  1994  IEEE  Time-Frequency  and  Time-Scale  Symp.,  Philadelphia,  PA,  Oct. 
1994. 

5.  Plenary  Speaker  at  the  1994  IEEE  Conf.  on  Acoust.  Speech  and  Signal  Proc.,  Adelaide,  Australia,  April 
1994. 

5.  Publications  Partially  Supported  by  Grant  AF/F49620-94-1-0461 

Book  Chapters: 

1.  “Issues  in  multimedia  databases:  Coding  for  content-based  image  retrieval  and  digital  copyright 
protection,”  M.  D.  Swanson  and  A.  H.  Tewfik,  in  The  Mathematics  of  Information  Coding,  Extraction  and 
Distribution,  G.  Cybenko,  D.  O’Leaiy  and  J.  Rissanen,  Eds.,  Springer-Verlag,  1998. 


2.  “Multiresolution  and  Object  Based  Video  Watermarking  Using  Perceptual  Models,”  in  Wavelet,  Subband 
And  Block  Transforms  In  Communications  And  Multimedia,  Ali  N.  Akansu  and  Michael  J.  Medley,  Eds., 
Kluwer  Academic  Publishers,  1998. 

Journal  Publications: 

1 .  “Multiresolution  Video  Watermarking  using  Perceptual  Models  and  Scene  Segmentation,”  M. 

Swanson,  B.  Zhu,  A.  H.  Tewfik  and  B.  Chau,  IEEE  J.  on  Selected  Areas  in  Comm.,  vol.  1 6,  no.  4,  pp. 
540-550,  May  1998. 

2.  “Robust  Audio  Watermarking  Using  Perceptual  Masking,”  M.  Swanson,  B.  Zhu,  A.  H.  Tewfik  and  L. 
Boney,  Signal  Processing,  vol.  66,  no.  3,  pp.  337-355,  May  1 998. 

3.  “Multimedia  Interaction,  Personalization  and  Protection:  Data  Embedding  and  Watermarking 
Technologies,”  M.  Swanson,  M.  Kobayashi  and  A.  H.  Tewfik,  invited  paper,  Proc.  IEEE,  vol.  86,  no.  6, 
pp.  1064-1087,  June  1998. 

4.  “Fast  Progressively  Refined  Image  Retrieval,”  M.  Swanson  and  A.  H.  Tewfik,  invited  paper,  Journal  of 
Electronic  Imaging,  1998. 

5.  “Audio  coding  for  representation  in  MIDI,”  N.  Sieger  and  A.  H.  Tewfik,  to  appear  in  Journal  of  VLSI 
Signal  Proc.  Systems  for  Speech,  Image  and  Video  Tech.,  invited  paper,  1998. 

6.  “Arithmetic  Coding  with  Dual  Symbol  Sets  and  Its  Performance  Analysis, "Bin  Zhu  and  A.  H.  Tewfik, 
accepted  for  publication,  to  appear  in  IEEE  Trans,  on  Image  Proc.,  1999. 

Invited  Tutorial  Presentations 

1.  “Watermarking  for  Audio,  Image  and  Video,”  1998  IEEE  Int.  Conf.  on  Image  Processing,  Chicago, 
October  1998. 
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Invited  Conference  Publications: 


1.  “Transparent  Robust  Image  Watermarking,”  M.  Swanson,  B.  Zhu  and  A.  H.  Tewfik,  invited  paper,  in  Proc. 
1996  IEEEInt.  Conf  Image  Proc.,  Lausanne,  Switzerland,  Sept  1996. 

2.  “Audio  Coding:  An  Overview,”  A.  H.  Tewfik,  invited  paper,  in  Twenty-Sixth  Annual  IEEE  Communication 
Theory  Workshop,  Tucson,  Arizona,  April  1997. 

3.  “Audio  Watermarking  and  Data  Embedding-Current  State  of  the  Art,  Challenges  and  Future  Directions,” 
invited  paper,  in  Multimedia  and  Security  Workshop,  ACM  Multimedia ’98,  Bristol,  September  1998. 

Other  Conference  Publications: 

L  “Image  Coding  with  Mixed  Representations  and  Visual  Masking”  B.  Zhu,  A.  H.  Tewfik  and  O.  Gerek,  in  Proc. 
of  the  1995  IEEE  Conf  on  Acoust.  Speech  and  Signal  Proc.,  Detroit,  MI,  May  1995. 

2.  “Image  Coding  with  Wavelet  Representations,  Edge  Information  and  Visual  Masking,”  B.  Zhu,  A.  H.  Tewfik, 
M.  A.  Colestock,  O.  N.  Gerek  and  A.  E.  Cetin,  1995  IEEE  Int.  Conf  Image  Proc.,  Washington,  D.C.,  Oct. 
1995. 

3.  “Image  Coding  for  Content  Based  Retrieval,”  M.  D.  Swanson,  S.  Hosur  and  A.  H.  Tewfik,  1996  Visual  Comm, 
and  Image  Proc.  (VCIP  ’96),  Orlando,  FI,  March  1996. 

4.  “Coding  for  Content-Based  Retrieval”,  M.  Swanson  and  A.  H.  Tewfik,  in  Proc.  of  the  1996  IEEE  Conf.  on 
Acoust.  Speech  and  Signal  Proc.,  Atlanta,  GA,  May  1996. 

5.  “Digital  watermarks  for  audio  signals,”  L.  Boney,  A.  H.  Tewfik  and  K.  Hamdy,  in  Proc.  IEEE  Multimedia 
Conf,  Hiroshima,  Japan,  June  1996. 

6.  “Digital  watermarks  for  audio  signals,”  L.  Boney,  A.  H.  Tewfik  and  K.  Hamdy,  in  Proc.  of  the  VII  European 
Signal  Proc.  Conf.  ( Eusipco-96 ).,  Trieste,  Italy,  Sept  1996. 

7.  “Robust  Data  hiding  for  images,”  M.  Swanson,  B.  Zhu  and  A.  H.  Tewfik,  in  Proc.  1996  IEEE  DSP  Workshop, 
Loen,  Norway,  Sept.  1996. 

8.  “Transparent  Robust  Authentication  and  Distortion  Measurement  Technique  for  Images,”  B.  Zhu,  M.  Swanson 
and  A.  H.  Tewfik,  in  Proc.  1996  IEEE  DSP  Workshop,  Loen,  Norway,  Sept.  1996. 

9.  “Dual  Set  Arithmetic  Coding  and  its  Application  in  Image  Coding,”  B.  Zhu,  E.  Yang  and  A.  H.  Tewfik,  in 
Proc.  of  the  VII European  Signal  Proc.  Conf.  (Eusipco-96).,  Trieste,  Italy,  Sept  1996. 

10. “Embedded  object  dictionaries  for  image  database  browsing  and  searching,”  M.  Swanson  and  A.  H.  Tewfik, 
1996  IEEEInt.  Conf  Image  Proc.,  Lausanne,  Switzerland,  Sept.  1996. 

1 1  “Combined  Audio  and  Visual  Streams  Analysis  for  Video  Sequence  Segmentation,”  J.  Nam  and  A.  H.  Tewfik, 
in  Proc.  of  the  1997  IEEE  Conf  on  Acoust.  Speech  and  Signal  Proc.,  Munich,  Germany,  April  1997. 

12.  ‘Time-Scale  Modification  of  Audio  Signals  with  Combined  Harmonic  and  Wavelet  Representations,”  K. 
Hamdy,  A.  H.  Tewfik,  T.  Cheng  and  S.  Takagi,  in  Proc.  of  the  1997  IEEE  Conf.  on  Acoust.  Speech  and  Signal 
Proc.,  Munich,  Germany,  April  1997. 

13. “Audio  coding  for  conversion  to  MIDI,”  N.  J.  Sieger  and  A.  H.  Tewfik,  in  Proc.  of  the  1997  IEEE  Workshop 
on  Multidimensional  Signal  Processing,  Princeton,  NJ,  June  1997. 
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14.  “Object  Based  Transparent  Video  Watermarking,”  M.  D.  Swanson,  B.  Zhu,  B.  Chau  and  A.  H.  Tewfik,  in 
Proc.  of the  1997  IEEE  Workshop  on  Multidimensional  Signal  Processing ,  Princeton,  NJ,  June  1 997. 

15.  “Adaptive  Low  Power  Multimedia  Communications,”  Tse-Hua  Lan  and  A.  H.  Tewfik,  in  Proc.  of  the  1997 
IEEE  Workshop  on  Multidimensional  Signal  Processing,  Princeton,  NJ,  June  1997. 

16.  “Affine-Invariant  Multiresolution  Image  Retrieval  Using  B-Splines,”  M.  Swanson  and  A.  H.  Tewfik,  Proc . 
1997  IEEE InL  Conf  Image  Proc.,  Santa  Barbara,  CA,  Oct.  1997. 

17.  “Joint  Audio-Visual  Streams  Analysis  for  Hierarchical  Video  Shot,”  J.  Nam,  E.  Cetin  and  A.  H.  Tewfik,  Proc. 

1997  IEEE  Int.  Conf  Image  Proc.,  Santa  Barbara,  CA,  Oct  1 997. 

18  “Data  Hiding  for  Video-in-Video, ”  M.  Swanson,  B.  Zhu  and  A.  H.  Tewfik,  Proc.  1997  IEEE  Int.  Conf  Image 
Proc.,  Santa  Barbara,  CA,  Oct.  1997. 

19.  “Multiresolution  Video  Watermarking  Using  Perceptual  Models  and  Scene  Segmentation,”  M.  D.  Swanson, 
B.  Zhu,  B.  Chau  and  A.  H.  Tewfik,  Proc.  1997  IEEE  Int.  Conf.  Image  Proc.,  Santa  Barbara,  CA,  Oct.  1 997. 

20.  “Eigen-Image  Based  Video  Segmentation  and  Indexing,”  K.  Han  and  A.  H.  Tewfik,  Proc.  1997  IEEE  Int. 
Conf.  Image  Proc.,  Santa  Barbara,  CA,  Oct  1997. 

21 .  “Image  Coding  by  Folding,”  B.  Zhu  and  A.  H.  Tewfik,  Proc.  1997  IEEE  Int.  Conf.  Image  Proc.,  Santa 
Barbara,  CA,  Oct  1997. 

22.  “Motion-Based  Video  Object  Indexing  Using  Multiresolution  Analysis  ”  J.  Nam  and  A.  H.  Tewfik,  Proc. 

1998  Electronic  Imaging:  Science  and  Tech.,  San  Jose,  CA,  Jan.  1998. 

23.  “Blind  Identification  of  Single-Input  Multiple-Output  Pole-Zero  Systems,”  G.  Venkatesan,  M.  Kaveh,  A.  H. 
Tewfik  and  K.  Buckley,  Proc.  of  the  1998  IEEE  Conf  on  Acoust.  Speech  and  Signal  Proc.,  Seattle,  WA,  May 
1998. 

24.  “Progressive  Resolution  Motion  Indexing  of  Video  Object,”  J.  Nam  and  A.  H.  Tewfik,  Proc.  of  the  1 998 
IEEE  Conf  on  Acoust.  Speech  and  Signal  Proc.,  Seattle,  WA,  May  1998. 

25. “ Audio  Visual  Content  based  Violent  Scene  Characterization,”  J.  Nam,  M.  Alghoniemy  and  A.  H.  Tewfik, 

Proc.  1998  IEEE  Int.  Conf.  Image  Proc.,  Chicago,  IL,  Oct.  1998. 

26. “Fast  Polynomial  Regression  Transform  For  Video  Database,”  K.  J.  Han  and  A.  H.  Tewfik,  Proc.  1998  IEEE 
Int.  Conf.  Image  Proc.,  Chicago,  IL,  Oct.  1998. 

27.  “Power  Optimized  Mode  Selection  For  H.263  Video  Coding  And  Wireless  Communications,”  T.  Lan  and  A. 

H.  Tewfik,  Proc.  1998  IEEE  Int.  Conf.  Image  Proc.,  Chicago,  IL,  Oct.  1998. 

28.  “Low  Complexity  Dynamic  Region  and  Translational  Motion  Estimation  for  Video  Indexing,”  K.  J.  Han  and 
A.  H.  Tewfik,  1998  IEEE  Workshop  on  Multimedia  Signal  Processing,  Redondo  Beach,  CA,  Dec.  1 998. 

29.  “Unified  Framework  for  Source-Channel-Modulation  Coding  in  Low  Power  Multimedia  Wireless 
communications,”  T.  Lan  and  A.  H.  Tewfik,  1998  IEEE  Workshop  on  Multimedia  Signal  Processing,  Redondo 
Beach,  CA,  Dec.  1998. 
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6.  Interactions 


Distinguished  IEEE  Lecturer  Presentations 

1.  Purdue  Univ.,  Dept,  of  Electrical  Engineering  Colloquium,  “Multimedia  Signal  Processing,”  Nov. 

1997. 

2.  Rochester  NY  IEEE  Signal  Processing  Chapter,  “Multimedia  Signal  Processing,”  Nov.  1997. 

3.  General  Electric  Corporate  Research  Laboratory,  “Multimedia  Signal  Processing,”  Nov.  1 997. 

4.  Washington  State  University,  Dept  of  Electrical  Engineering  Colloquium,  “Multimedia  Signal 
Processing,”  March  1998. 

5.  NJIT,  Dept,  of  Electrical  Engineering  Colloquium,  “Multimedia  Signal  Processing,”  April  1 998. 

6.  University  of  Minnesota  and  IEEE  Signal  Processing  Chapter,  Twin  Cities,  “E-commerce, 
Multimedia  and  Signal  Processing,”  April  1 998. 

Other  Presentations 

1 .  University  of  Minnesota,  Physics  Colloquium,  “Wavelets:  Theory  and  Applications,”  Feb.  1 994. 

2.  ARPA,  Washington,  D.C.,  “Robust  Multiresolution  Integrated  Target  Sensing  And  Recognition,”  July 
1994. 

3.  NSA,  Washington,  D.C.,  “Wavelets:  Theory  and  Applications,”  Jan.  1995. 

4.  NRaD,  San  Diego,  CA,  “Adaptive  Waveform  Selection  for  High  Resolution  Range-Doppler  Radar 
Imaging,”  Feb.  1995. 

5.  Texas  Instruments,  Dallas,  TX,  “Signal  Processing  and  Multimedia,”  Jan.  1996. 

6.  IBM,  Tokyo,  Japan,  “Watermarking  Digital  Audio  and  Imagery,”  June  1996. 

7.  Tokyo  Univ,  Dept,  of  Math.,  “Wavelets:  Potentials  and  Limitations,”  June  1996 

8.  Bilkent  Univ.,  Ankara,  Turkey,  “Signal  Processing  and  Multimedia,”  July  1996. 

9.  NEC,  Princeton,  NJ,  “Digital  Watermarks  for  Audio  and  Images,”  Aug.  1996. 

10.  Northern  Illinois  University,  Dept,  of  Electrical  Engineering  Colloquium,  “Multimedia:  The  new 
Signal  Processing  Frontier,”  Oct.  1996. 

1 1 .  Drexel  Univ.,  Dept,  of  Electrical  Engineering  Colloquium,  “New  Vistas  in  Signal  Processing:  The 
Multimedia  Challenge,”  Nov.  1996. 

12.  Optical  Society  of  America,  Minneapolis  Chapter,  “Wavelets:  Theory  and  Applications,”  May  1 998. 
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7.  Patents 


The  following  patents  were  filed: 


Filed 

Seriel  No. 

Title 

Inventors 

Ahmed  H.  Tewfik 

8/27/97 

918,122 

Method  and  Apparatus  for  Embedding  Data, 

Including  Watermarks,  in  Human  Perceptible  Images 

Mitchell  D.  Swanson 
Bin  Zhu 

Ahmed  H.  Tewfik 

8/27/97 

918,125 

Method  and  Apparatus  for  Video  Watermarking 

Mitchell  D.  Swanson 
Bin  Zhu 

Ahmed  H.  Tewfik 

8/27/97 

918,126 

Digital  Watermarking  to  Resolve  Claims  of  Multiple 
Ownership 

Mitchell  D.  Swanson 
Bin  Zhu 

Ahmed  H.  Tewfik 

8/27/97 

918,891 

Method  and  Apparatus  for  Embedding  Data, 

Including  Human  Perceptible  Sounds 

Mitchell  D.  Swanson 
Bin  Zhu 

Laurence  M.  Boney 

8/27/97 

921,931 

Method  and  Apparatus  for  Scene-based  Video 
Watermarking 

Ahmed  H.  Tewfik 
Mitchell  D.  Swanson 
Bin  Zhu 
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